Nov 03 22:21:54.219123 ip-10-0-139-173 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:21:54.619635 ip-10-0-139-173 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:54.619635 ip-10-0-139-173 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:21:54.619635 ip-10-0-139-173 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:54.619635 ip-10-0-139-173 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:54.625320 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625238 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.625320 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625320 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.625320 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625324 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625327 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625330 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625333 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625336 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625339 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625342 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625344 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625347 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625350 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625352 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625355 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625357 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625360 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625363 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625366 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625368 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625371 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625373 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625376 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.625416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625378 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625381 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625383 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625386 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625392 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625397 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625400 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625402 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625405 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625407 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625410 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625412 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625415 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625418 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625421 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625424 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625426 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625429 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625432 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625435 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.625948 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625437 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625440 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625443 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625445 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625448 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625450 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625453 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625455 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625458 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625460 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625465 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625470 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625473 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625477 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625480 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625483 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625487 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625490 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625493 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.626428 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625495 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625498 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625501 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625503 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625509 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625512 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625515 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625517 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625520 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625523 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625526 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625528 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625531 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625534 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625537 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625540 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625543 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625546 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625548 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.626884 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625551 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625554 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625556 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625559 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625562 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625564 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625567 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625570 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625574 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625577 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625582 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625585 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625588 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625591 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625594 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625596 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625599 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625602 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625605 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.627396 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625607 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625610 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625613 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625615 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625618 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.625620 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626489 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626497 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626500 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626502 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626506 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626508 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626511 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626514 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626517 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626520 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626522 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626525 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626527 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626530 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.627849 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626533 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626536 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626538 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626541 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626545 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626548 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626551 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626553 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626556 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626559 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626562 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626564 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626567 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626570 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626572 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626575 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626579 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626581 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626584 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626586 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.628343 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626589 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626592 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626594 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626597 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626600 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626603 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626605 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626608 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626611 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626613 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626616 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626618 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626621 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626624 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626627 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626629 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626631 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626635 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626639 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.628838 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626643 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626649 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626652 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626655 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626658 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626661 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626663 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626666 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626668 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626685 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626689 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626692 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626695 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626698 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626701 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626703 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626706 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626710 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626712 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626717 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.629300 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626721 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626724 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626728 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626731 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626734 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626737 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626739 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626742 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626745 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626747 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626750 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626753 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626756 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626760 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626762 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626765 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626767 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626770 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626773 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626775 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.629830 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626778 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626781 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626783 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626799 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626802 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626805 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626808 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626811 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626814 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626818 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626821 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626824 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.626827 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628092 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628098 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628102 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628105 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628108 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628111 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.630302 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628116 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628119 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628122 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628125 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628128 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628131 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628134 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628137 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628139 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628142 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628145 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628148 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628150 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628153 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628155 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628158 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628160 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628163 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628165 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628168 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.630781 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628170 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628173 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628176 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628179 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628182 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628184 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628187 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628189 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628192 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628194 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628197 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628199 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628202 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628204 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628207 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628210 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628213 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628216 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628218 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628221 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.631284 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628223 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628226 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628228 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628231 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628234 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628237 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628240 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628242 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628245 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628247 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628250 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628252 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628255 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628258 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628260 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628263 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628266 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628268 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628271 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.631771 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628273 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628276 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628279 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628281 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628284 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628286 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628289 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628292 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628295 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628297 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628300 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628303 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628306 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628308 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628310 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628313 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628316 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628318 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628321 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.632242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628323 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628326 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628328 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628331 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628333 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628336 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628339 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628343 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628347 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628351 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628355 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628359 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628363 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628366 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628369 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628372 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628374 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628377 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628380 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.632685 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628382 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.628385 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629197 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629207 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629213 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629220 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629224 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629228 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629232 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629240 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629243 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629246 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629249 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629253 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629256 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629259 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629261 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629264 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629267 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629270 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629275 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629278 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629281 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629285 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:21:54.633145 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629288 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629293 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629296 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629299 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629302 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629306 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629309 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629312 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629316 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629321 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629326 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629329 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629332 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629335 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629338 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629342 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629346 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629349 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629352 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629355 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629358 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629363 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629367 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629370 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629373 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:21:54.633714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629376 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629379 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629383 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629386 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629389 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629391 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629394 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629399 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629402 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629405 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629408 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629411 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629415 2578 flags.go:64] FLAG: --help="false" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629418 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629421 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629424 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629428 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629432 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629435 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629438 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629442 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629445 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629448 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:21:54.634396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629451 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629455 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629458 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629461 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629464 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629467 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629470 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629473 2578 flags.go:64] FLAG: --lock-file="" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629476 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629479 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629482 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629488 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629491 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629494 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629496 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629499 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629503 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629506 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629509 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629513 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629516 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629520 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629523 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629527 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629530 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:21:54.635066 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629534 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629538 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629541 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629543 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629551 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629554 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629557 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629561 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629564 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629567 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629571 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629574 2578 flags.go:64] FLAG: --port="10250" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629577 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629580 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d594187a7cebf8f0" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629583 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629586 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629589 2578 flags.go:64] FLAG: --register-node="true" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629592 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629598 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629601 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629604 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629607 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629611 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629614 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629617 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629620 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:21:54.635727 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629624 2578 flags.go:64] FLAG: --runonce="false" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629627 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629630 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629633 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629636 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629639 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629644 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629647 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629650 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629653 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629656 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629659 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629661 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629665 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629668 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629671 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629677 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629680 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629683 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629686 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629689 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629692 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629695 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629698 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629703 2578 flags.go:64] FLAG: --v="2" Nov 03 22:21:54.636383 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629708 2578 flags.go:64] FLAG: --version="false" Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629712 2578 flags.go:64] FLAG: --vmodule="" Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629716 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.629720 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629905 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629910 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629913 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629918 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629921 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629924 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629927 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629930 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629933 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629936 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629939 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629941 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629944 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629947 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629949 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629952 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.637001 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629954 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629958 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629961 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629964 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629966 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629969 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629971 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629974 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629977 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629979 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629982 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629986 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629988 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629991 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629994 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.629996 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630000 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630004 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630007 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.637481 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630011 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630014 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630017 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630019 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630022 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630024 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630027 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630030 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630032 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630035 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630037 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630040 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630042 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630045 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630048 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630051 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630053 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630056 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630059 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630061 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.638022 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630064 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630066 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630069 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630071 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630075 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630077 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630080 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630082 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630085 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630087 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630090 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630093 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630095 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630097 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630100 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630102 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630105 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630107 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630110 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.638504 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630113 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630115 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630118 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630121 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630123 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630126 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630128 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630131 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630134 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630136 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630139 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630142 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630144 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630147 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630149 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630152 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630155 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630158 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630161 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630164 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.639003 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630166 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630169 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630172 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630174 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630177 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630179 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630182 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630184 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630187 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630189 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630192 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630194 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.630220 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630260 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630264 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630267 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.639473 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630270 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630273 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630276 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630278 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630281 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630283 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630286 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630289 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630292 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630295 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630298 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630302 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630306 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630310 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630313 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630316 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630318 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630321 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630324 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.639878 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630326 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630329 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630332 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630334 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630337 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630339 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630342 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630344 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630358 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630362 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630364 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630367 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630370 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630372 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630375 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630378 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630380 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630383 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630386 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.640327 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630388 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630392 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630396 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630399 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630402 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630405 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630407 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630411 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630414 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630417 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630419 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630422 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630424 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630427 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630429 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630432 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630435 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630437 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630441 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.640764 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630443 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630446 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630450 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630452 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630455 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630457 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630460 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630463 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630466 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630468 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630471 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630473 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630476 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630479 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630481 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630484 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630487 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630489 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630492 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.641242 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630499 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630501 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630505 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630507 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630510 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630512 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630515 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630518 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630520 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630523 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630526 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630528 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630531 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630533 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630537 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630540 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630543 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630545 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630548 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630551 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.641728 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630553 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630556 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630559 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630561 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630564 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.630566 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.637608 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.637622 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.637634 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.637639 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637693 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637698 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637701 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637704 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637707 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637710 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637712 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637716 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637718 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.642270 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637721 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637723 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637726 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637729 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637732 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637735 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637738 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637740 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637743 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637747 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637751 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637755 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637758 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637760 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637763 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637766 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637769 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637772 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.642726 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637775 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637778 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637781 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637783 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637803 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637807 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637810 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637812 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637815 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637818 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637821 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637823 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637826 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637828 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637831 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637834 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637837 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637839 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637842 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637845 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.643174 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637847 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637850 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637853 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637855 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637858 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637860 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637863 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637865 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637868 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637870 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637874 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637876 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637879 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637882 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637885 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637887 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637890 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637893 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637896 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637899 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.643634 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637901 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637904 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637906 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637909 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637912 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637914 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637917 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637919 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637922 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637925 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637927 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637930 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637932 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637935 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637938 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637940 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637942 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637945 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637948 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637950 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.644145 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637954 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637958 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637961 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637965 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637971 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637974 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637976 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637979 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637982 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637985 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637988 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637990 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637993 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637995 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.637998 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638000 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638003 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.644665 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638005 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.638033 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638160 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638165 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638168 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638171 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638174 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638176 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638179 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638182 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638184 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638187 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638190 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638192 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638195 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638197 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.645097 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638200 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638203 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638206 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638208 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638211 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638213 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638216 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638218 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638221 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638224 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638227 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638229 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638232 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638234 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638237 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638242 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638245 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638247 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638249 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.645463 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638252 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638254 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638257 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638262 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638265 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638267 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638270 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638273 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638275 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638278 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638280 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638283 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638287 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638290 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638294 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638298 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638302 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638305 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.646089 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638310 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638312 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638315 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638318 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638321 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638323 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638326 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638328 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638331 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638333 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638336 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638338 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638341 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638344 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638346 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638357 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638360 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638363 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638365 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.646566 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638368 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638371 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638373 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638376 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638379 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638381 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638384 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638387 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638389 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638392 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638395 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638397 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638400 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638402 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638405 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638408 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638410 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638413 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638416 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.647335 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638418 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638421 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638423 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638426 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638428 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638431 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638433 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638436 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638438 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638441 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638444 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638447 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638450 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638452 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.638475 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638506 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:54.648104 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638511 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638514 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638517 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638520 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638523 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638526 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638528 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638531 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638533 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638536 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638539 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638541 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638544 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638547 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638550 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638552 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638555 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638557 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638560 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:54.648537 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638562 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638565 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638568 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638570 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638573 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638575 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638578 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638581 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638583 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638586 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638588 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638591 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638593 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638596 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638598 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638601 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638609 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638612 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638615 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638618 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:54.649086 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638620 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638623 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638625 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638628 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638630 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638633 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638636 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638638 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638641 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638644 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638646 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638649 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638651 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638654 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638656 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638659 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638661 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638664 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638669 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638671 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:54.649675 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638674 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638677 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638679 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638682 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638684 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638687 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638689 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638692 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638695 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638697 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638700 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638702 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638705 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638707 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638710 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638712 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638715 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638717 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638720 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638722 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:54.650220 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638725 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638728 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638730 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638732 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638735 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638739 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638743 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638747 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638751 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638754 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638760 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638763 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638765 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638768 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638770 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638773 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638775 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638778 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638782 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:54.650693 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638798 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638801 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638804 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638807 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638809 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638812 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:54.638814 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.638865 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.641073 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.642085 2578 server.go:1014] "Starting client certificate rotation" Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.642176 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.642207 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.644336 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:21:54.651155 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.649258 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:21:54.661284 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.661256 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:21:54.664885 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.664867 2578 log.go:25] "Validated CRI v1 image API" Nov 03 22:21:54.665607 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.665583 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:21:54.669985 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.669966 2578 fs.go:135] Filesystem UUIDs: map[65fd09cf-bf74-4d6a-b9b2-90e88c405ca0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eaca8f28-a109-42f8-b0eb-4f0ab6413648:/dev/nvme0n1p4] Nov 03 22:21:54.670034 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.669986 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:21:54.675231 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.675118 2578 manager.go:217] Machine: {Timestamp:2026-11-03 22:21:54.673720467 +0000 UTC m=+0.339830630 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102478 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b87875cc38baeb20897ee2bb7ffef SystemUUID:ec2b8787-5cc3-8bae-b208-97ee2bb7ffef BootID:352bf077-e266-41ff-9fcc-9d5a91a40049 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f7:ad:62:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f7:ad:62:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:40:0a:e7:9e:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:21:54.675928 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.675915 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:21:54.676015 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.676003 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:21:54.678030 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.678004 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:21:54.678162 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.678030 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-173.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:21:54.678229 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.678165 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:21:54.678229 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.678173 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:21:54.678229 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.678220 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:21:54.678307 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.678240 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:21:54.679453 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.679442 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:21:54.683639 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.683627 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:21:54.683769 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.683759 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:21:54.684660 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.684644 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:21:54.684695 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.684666 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:21:54.684695 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.684678 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:21:54.684695 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.684689 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:21:54.684815 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.684697 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:21:54.685045 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.685034 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:21:54.687451 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.687434 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:21:54.694751 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.694737 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:21:54.695759 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.695745 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:21:54.695819 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.695782 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:21:54.696018 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.696007 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:21:54.696050 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.696026 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:21:54.696050 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.696036 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:21:54.696050 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.696044 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:21:54.696132 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.696052 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:21:54.696132 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.696074 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:21:54.697043 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.697033 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:21:54.697075 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.697048 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:21:54.697075 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.697058 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:21:54.697075 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.697066 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:21:54.697156 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.697083 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:21:54.697156 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.697098 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:21:54.698334 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.698324 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:21:54.698375 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.698337 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:21:54.702990 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.702976 2578 server.go:1263] "Started kubelet" Nov 03 22:21:54.703099 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.703073 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:21:54.703165 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.703070 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:21:54.704508 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.704488 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:21:54.705101 ip-10-0-139-173 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:21:54.706970 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.706951 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:21:54.709441 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.709422 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:21:54.711421 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.711404 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:21:54.715158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.715141 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:21:54.715242 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.715161 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:21:54.715402 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.715381 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:54.715986 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.715964 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:21:54.716093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.715972 2578 factory.go:55] Registering systemd factory Nov 03 22:21:54.716093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716004 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:21:54.716093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.715973 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:21:54.716093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716051 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:21:54.716275 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716128 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:21:54.716275 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716138 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:21:54.716552 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716534 2578 factory.go:153] Registering CRI-O factory Nov 03 22:21:54.716552 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716552 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 22:21:54.716697 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716600 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:21:54.716697 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716618 2578 factory.go:103] Registering Raw factory Nov 03 22:21:54.716697 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.716634 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 22:21:54.717743 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.717728 2578 manager.go:319] Starting recovery of all containers Nov 03 22:21:54.730745 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.730727 2578 manager.go:324] Recovery completed Nov 03 22:21:54.734747 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.734734 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:54.736708 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.736686 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:54.736775 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.736732 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:54.736775 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.736747 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:54.737388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.737374 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:21:54.737388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.737387 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:21:54.737514 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.737392 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:54.737514 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.737406 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:21:54.738246 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.738230 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:54.738318 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.738270 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:54.738318 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.738285 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:54.739570 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.739558 2578 policy_none.go:50] "Start" Nov 03 22:21:54.739625 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.739578 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:21:54.739625 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.739589 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:21:54.741564 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.741552 2578 policy_none.go:44] "Start" Nov 03 22:21:54.791401 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.784449 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:21:54.791401 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.785695 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:21:54.791401 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.785717 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:21:54.791401 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.785753 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:21:54.791401 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.785808 2578 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:21:54.815747 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.815721 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:54.839597 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.839565 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-173.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:21:54.839720 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.839593 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-173.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:21:54.840838 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.839572 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-173.us-west-2.compute.internal.18e4a921e8ac94b3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-173.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-173.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:21:54.702939315 +0000 UTC m=+0.369049479,LastTimestamp:2026-11-03 22:21:54.702939315 +0000 UTC m=+0.369049479,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-173.us-west-2.compute.internal,}" Nov 03 22:21:54.877413 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.877348 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:21:54.879888 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.879683 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 22:21:54.879974 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.879944 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:21:54.879974 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.879955 2578 server.go:87] "Starting device plugin registration server" Nov 03 22:21:54.880231 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.880218 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:21:54.880286 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.880232 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:21:54.880364 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.880337 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:21:54.880426 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.880420 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:21:54.880477 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.880427 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:21:54.881049 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.881027 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:21:54.881140 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.881085 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:54.886644 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.886607 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal"] Nov 03 22:21:54.886708 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.886694 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:54.887477 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.887459 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:54.887545 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.887492 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:54.887545 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.887503 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:54.888065 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.888045 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-173.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 22:21:54.889416 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.889401 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:54.890089 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890076 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:54.890146 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890102 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:54.890146 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890112 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:54.890263 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:54.890301 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890282 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:54.890991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890945 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:54.890991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890973 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:54.890991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.890984 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:54.891901 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.891886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:54.891967 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.891910 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:54.892621 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.892601 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:54.892693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.892628 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:54.892693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.892643 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:54.908893 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.908870 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:21:54.908985 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.908952 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjq6k" Nov 03 22:21:54.924121 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.924099 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:54.926823 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.926806 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:54.955610 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.955577 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 22:21:54.955610 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.955591 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 22:21:54.955825 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.955559 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-173.us-west-2.compute.internal.18e4a921eaafdd78 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-173.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-173.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-173.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:21:54.736708984 +0000 UTC m=+0.402819155,LastTimestamp:2026-11-03 22:21:54.736708984 +0000 UTC m=+0.402819155,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-173.us-west-2.compute.internal,}" Nov 03 22:21:54.955825 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:54.955657 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:21:54.980377 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.980335 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:54.981365 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.981347 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:54.981443 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.981378 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:54.981443 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.981388 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:54.981443 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:54.981417 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.017817 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.017777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.017959 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.017822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.017959 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.017848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3dade66e2d9f1769c6a8531aeeaf63ad-config\") pod \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"3dade66e2d9f1769c6a8531aeeaf63ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.047431 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.047406 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.047507 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.047437 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-173.us-west-2.compute.internal\": node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.118005 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.117975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.118158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.118007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.118158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.118037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3dade66e2d9f1769c6a8531aeeaf63ad-config\") pod \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"3dade66e2d9f1769c6a8531aeeaf63ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.118158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.118092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.118158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.118091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3dade66e2d9f1769c6a8531aeeaf63ad-config\") pod \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"3dade66e2d9f1769c6a8531aeeaf63ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.118158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.118097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.157234 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.157172 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.229569 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.229542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.231707 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.231692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:55.257840 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.257818 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.360188 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.360155 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.463451 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.463373 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.565395 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.565361 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.642565 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.642542 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:21:55.643047 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.642689 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a16baf3e91ce6443fadccbde99d65a93-20d6be8647d4ee72.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-173.us-west-2.compute.internal.18e4a921eaafdd78\": read tcp 10.0.139.173:43208->52.15.70.29:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-173.us-west-2.compute.internal.18e4a921eaafdd78 default 4866 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-173.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-173.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-173.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:21:54 +0000 UTC,LastTimestamp:2026-11-03 22:21:54.890088451 +0000 UTC m=+0.556198614,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-173.us-west-2.compute.internal,}" Nov 03 22:21:55.666112 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.666089 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.769478 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.769423 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.870049 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.870008 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:55.914714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.914662 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:16:55 +0000 UTC" deadline="2028-05-26 11:47:02.863326702 +0000 UTC" Nov 03 22:21:55.914714 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.914704 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13669h25m6.948626801s" Nov 03 22:21:55.945149 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.945125 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:55.946250 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.946224 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:55.946356 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.946267 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:55.946356 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:55.946282 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:55.971027 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:55.971003 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:56.048016 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.047947 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:56.071945 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.071922 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:56.172293 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.172257 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:56.273315 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.273271 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:21:56.352171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.352145 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:56.415467 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.415423 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:56.431890 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:56.431847 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dade66e2d9f1769c6a8531aeeaf63ad.slice/crio-c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70 WatchSource:0}: Error finding container c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70: Status 404 returned error can't find the container with id c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70 Nov 03 22:21:56.435134 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.435111 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:56.439225 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.439207 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:21:56.478713 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:56.478687 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c43c551f0a2ae66c9cba55c176ecea.slice/crio-03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e WatchSource:0}: Error finding container 03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e: Status 404 returned error can't find the container with id 03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e Nov 03 22:21:56.484352 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.484334 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:21:56.486226 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.486212 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:21:56.549726 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.549708 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:21:56.607090 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.607064 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:56.686437 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.686411 2578 apiserver.go:52] "Watching apiserver" Nov 03 22:21:56.717977 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.717949 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:21:56.789975 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.789925 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"d4c43c551f0a2ae66c9cba55c176ecea","Type":"ContainerStarted","Data":"03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e"} Nov 03 22:21:56.791126 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.791100 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"3dade66e2d9f1769c6a8531aeeaf63ad","Type":"ContainerStarted","Data":"c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70"} Nov 03 22:21:56.799496 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.799475 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:21:56.799802 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.799786 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92","openshift-dns/node-resolver-fqr6l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-rcrd8","openshift-network-diagnostics/network-check-target-mzwbr","openshift-network-operator/iptables-alerter-qsnsn","kube-system/global-pull-secret-syncer-c9btg","kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-zg9h5","openshift-image-registry/node-ca-2g542","openshift-multus/multus-additional-cni-plugins-jqclr","openshift-multus/multus-pjjqw","openshift-ovn-kubernetes/ovnkube-node-rhx44","kube-system/konnectivity-agent-pvpqf"] Nov 03 22:21:56.803654 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.803636 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.805922 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.805901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.808140 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.808119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.810573 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.810360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:56.810573 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.810455 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:21:56.812663 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.812644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.815027 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.815009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.817404 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.817379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.820101 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.820077 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.823312 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.823295 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:21:56.824138 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.824111 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:21:56.826523 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.826506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.828967 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.828949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:56.829040 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.829002 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:21:56.830041 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:56.830105 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.830105 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxd8q\" (UniqueName: \"kubernetes.io/projected/1d7234dd-7b73-427d-b0e1-afd778bd2028-kube-api-access-fxd8q\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.830179 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-cni-binary-copy\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.830179 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-socket-dir-parent\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.830250 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vq5n\" (UniqueName: \"kubernetes.io/projected/de375da1-1b02-4796-ad49-2062bc5bc74b-kube-api-access-8vq5n\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:56.830250 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.830250 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-sys\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.830250 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgpc9\" (UniqueName: \"kubernetes.io/projected/1f23b7eb-3f0e-4215-a42c-c20c976900dc-kube-api-access-cgpc9\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.830403 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.830403 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-modprobe-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.830403 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-conf\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.830509 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qs7k\" (UniqueName: \"kubernetes.io/projected/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-kube-api-access-5qs7k\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.830509 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.830509 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.830509 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/458febd7-7dcd-4ef1-9b8b-27037557b753-host-slash\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.830773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysconfig\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.830773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-lib-modules\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.830773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-system-cni-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.830773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.830773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-system-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.830773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-k8s-cni-cncf-io\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.830773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-conf-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-multus-certs\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-kubernetes\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-run\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-var-lib-kubelet\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.830973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-tuned\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvnh\" (UniqueName: \"kubernetes.io/projected/3714e210-a948-4ba6-a174-0fc8eb8e06db-kube-api-access-bzvnh\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/458febd7-7dcd-4ef1-9b8b-27037557b753-iptables-alerter-script\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.831106 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjrh\" (UniqueName: \"kubernetes.io/projected/458febd7-7dcd-4ef1-9b8b-27037557b753-kube-api-access-9wjrh\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-systemd\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-serviceca\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cnibin\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-cnibin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-netns\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-kubelet\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-hostroot\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-daemon-config\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-host\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-host\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.831433 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:56.832774 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7234dd-7b73-427d-b0e1-afd778bd2028-tmp-dir\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.832774 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-etc-kubernetes\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.832774 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8xp\" (UniqueName: \"kubernetes.io/projected/a9b5b425-585c-4fc9-bc11-2510c7035c70-kube-api-access-sx8xp\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.832774 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-tmp\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.832774 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d7234dd-7b73-427d-b0e1-afd778bd2028-hosts-file\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.832774 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-bin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.832774 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.831658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-multus\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.861000 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.860933 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:21:56.861000 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.860931 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:21:56.861000 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.860984 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:21:56.861209 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.861088 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kfchj\"" Nov 03 22:21:56.862765 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.862745 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6sdx\"" Nov 03 22:21:56.862895 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.862864 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:21:56.862965 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.862899 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:21:56.862965 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.862904 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:21:56.862965 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.862940 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:21:56.863241 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.863191 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:21:56.864825 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.864811 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:21:56.864979 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.864963 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ljhlr\"" Nov 03 22:21:56.869685 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.869668 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:21:56.869784 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.869771 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nh9f8\"" Nov 03 22:21:56.869843 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.869829 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:21:56.877514 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877491 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:56.877514 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877501 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:56.877657 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877643 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:56.877701 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877655 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:56.877743 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877713 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:21:56.877743 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877735 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hvj2k\"" Nov 03 22:21:56.877840 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877792 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kccnl\"" Nov 03 22:21:56.877892 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877837 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:21:56.877892 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877842 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:21:56.877892 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.877876 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mdpv5\"" Nov 03 22:21:56.913093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.913063 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:21:56.913213 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.913134 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g6szk" Nov 03 22:21:56.916796 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.916777 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:21:56.931921 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.931893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-host\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.932026 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.931937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-dbus\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:56.932026 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.931969 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-host\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.932026 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.931985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.932164 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d7234dd-7b73-427d-b0e1-afd778bd2028-hosts-file\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.932164 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-multus\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.932164 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-netns\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.932164 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-multus\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.932164 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.932363 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-registration-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.932363 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d7234dd-7b73-427d-b0e1-afd778bd2028-hosts-file\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.932363 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.932363 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxd8q\" (UniqueName: \"kubernetes.io/projected/1d7234dd-7b73-427d-b0e1-afd778bd2028-kube-api-access-fxd8q\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.932363 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-script-lib\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.932621 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-device-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.932621 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932446 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.932925 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vq5n\" (UniqueName: \"kubernetes.io/projected/de375da1-1b02-4796-ad49-2062bc5bc74b-kube-api-access-8vq5n\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:56.932969 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-sys\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933008 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgpc9\" (UniqueName: \"kubernetes.io/projected/1f23b7eb-3f0e-4215-a42c-c20c976900dc-kube-api-access-cgpc9\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.933008 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.932998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933092 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-env-overrides\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.933092 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-conf\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933092 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-sys\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933209 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qs7k\" (UniqueName: \"kubernetes.io/projected/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-kube-api-access-5qs7k\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.933209 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.933209 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.933209 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/458febd7-7dcd-4ef1-9b8b-27037557b753-host-slash\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.933209 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933207 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-conf\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a374ff-7146-47f5-bcbd-6c4dc710d580-agent-certs\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/458febd7-7dcd-4ef1-9b8b-27037557b753-host-slash\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933264 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-etc-kubernetes\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933296 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-etc-kubernetes\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysconfig\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-system-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-conf-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysconfig\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-multus-certs\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-slash\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933441 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-conf-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-bin\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-system-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-run\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-tuned\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjrh\" (UniqueName: \"kubernetes.io/projected/458febd7-7dcd-4ef1-9b8b-27037557b753-kube-api-access-9wjrh\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cnibin\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-cnibin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-run\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-kubelet\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-multus-certs\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933724 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-kubelet\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-hostroot\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cnibin\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-cnibin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-ovn\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724a681a-c911-439e-aa2c-904081578713-ovn-node-metrics-cert\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933820 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-hostroot\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.933938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-host\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7234dd-7b73-427d-b0e1-afd778bd2028-tmp-dir\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933986 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-config\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-socket-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-tmp\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-bin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-log-socket\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.933989 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-host\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-cni-binary-copy\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.934227 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-socket-dir-parent\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a374ff-7146-47f5-bcbd-6c4dc710d580-konnectivity-ca\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7234dd-7b73-427d-b0e1-afd778bd2028-tmp-dir\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934422 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-bin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934413 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:21:56.934511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-systemd-units\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:56.934492 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:21:57.434445784 +0000 UTC m=+3.100555952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-modprobe-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934541 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-socket-dir-parent\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-kubelet-config\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-netd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-modprobe-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-lib-modules\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-system-cni-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-system-cni-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.935388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-k8s-cni-cncf-io\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934879 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-lib-modules\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-var-lib-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.934994 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-cni-binary-copy\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935001 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-k8s-cni-cncf-io\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-node-log\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/add89f53-e47e-41de-8d0d-6fc06d826d29-kube-api-access-dstxj\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-kubernetes\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-var-lib-kubelet\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvnh\" (UniqueName: \"kubernetes.io/projected/3714e210-a948-4ba6-a174-0fc8eb8e06db-kube-api-access-bzvnh\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/458febd7-7dcd-4ef1-9b8b-27037557b753-iptables-alerter-script\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-kubernetes\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-var-lib-kubelet\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-daemon-config\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-sys-fs\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-systemd\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936171 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-serviceca\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-netns\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8xp\" (UniqueName: \"kubernetes.io/projected/a9b5b425-585c-4fc9-bc11-2510c7035c70-kube-api-access-sx8xp\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-kubelet\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-systemd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-netns\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-etc-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsqkb\" (UniqueName: \"kubernetes.io/projected/724a681a-c911-439e-aa2c-904081578713-kube-api-access-dsqkb\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-systemd\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-daemon-config\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.935818 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/458febd7-7dcd-4ef1-9b8b-27037557b753-iptables-alerter-script\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.936658 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.936111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-serviceca\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:56.937792 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.937776 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-tuned\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.938423 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.938408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-tmp\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.970654 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.970585 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.970706 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.970659 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.970997 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.971073 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hp7n8\"" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.971202 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:21:56.971282 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.971240 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:21:56.971748 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.971401 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ztkgs\"" Nov 03 22:21:56.971748 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.971464 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:21:56.971748 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.971576 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:21:56.999440 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.999414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjrh\" (UniqueName: \"kubernetes.io/projected/458febd7-7dcd-4ef1-9b8b-27037557b753-kube-api-access-9wjrh\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:56.999440 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.999418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxd8q\" (UniqueName: \"kubernetes.io/projected/1d7234dd-7b73-427d-b0e1-afd778bd2028-kube-api-access-fxd8q\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:56.999440 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:56.999436 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vq5n\" (UniqueName: \"kubernetes.io/projected/de375da1-1b02-4796-ad49-2062bc5bc74b-kube-api-access-8vq5n\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:57.000287 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.000270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgpc9\" (UniqueName: \"kubernetes.io/projected/1f23b7eb-3f0e-4215-a42c-c20c976900dc-kube-api-access-cgpc9\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:57.000355 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.000338 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qs7k\" (UniqueName: \"kubernetes.io/projected/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-kube-api-access-5qs7k\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:57.004789 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.004766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvnh\" (UniqueName: \"kubernetes.io/projected/3714e210-a948-4ba6-a174-0fc8eb8e06db-kube-api-access-bzvnh\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:57.005067 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.005051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8xp\" (UniqueName: \"kubernetes.io/projected/a9b5b425-585c-4fc9-bc11-2510c7035c70-kube-api-access-sx8xp\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:21:57.036326 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-kubelet\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-systemd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-etc-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqkb\" (UniqueName: \"kubernetes.io/projected/724a681a-c911-439e-aa2c-904081578713-kube-api-access-dsqkb\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-dbus\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:57.036447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-netns\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-etc-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036446 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-systemd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-kubelet\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-netns\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036527 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-registration-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-dbus\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-script-lib\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-device-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-registration-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-env-overrides\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a374ff-7146-47f5-bcbd-6c4dc710d580-agent-certs\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036672 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-device-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-slash\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.036752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-bin\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-ovn\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724a681a-c911-439e-aa2c-904081578713-ovn-node-metrics-cert\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-config\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-slash\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-socket-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-ovn\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-bin\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-log-socket\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-socket-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.036983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a374ff-7146-47f5-bcbd-6c4dc710d580-konnectivity-ca\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-systemd-units\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-kubelet-config\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-netd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.037111 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.037582 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-var-lib-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-node-log\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/add89f53-e47e-41de-8d0d-6fc06d826d29-kube-api-access-dstxj\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-env-overrides\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-sys-fs\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037324 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-sys-fs\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037336 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-script-lib\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037350 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-log-socket\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-config\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-netd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037398 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-systemd-units\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-kubelet-config\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037446 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-node-log\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-var-lib-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.038208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.038695 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.037627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a374ff-7146-47f5-bcbd-6c4dc710d580-konnectivity-ca\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:57.038695 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.038027 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret podName:06b3993b-eaa8-4649-9c38-2b9bb7a22498 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:57.538005493 +0000 UTC m=+3.204115646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret") pod "global-pull-secret-syncer-c9btg" (UID: "06b3993b-eaa8-4649-9c38-2b9bb7a22498") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:57.039179 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.039162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724a681a-c911-439e-aa2c-904081578713-ovn-node-metrics-cert\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.040144 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.040129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a374ff-7146-47f5-bcbd-6c4dc710d580-agent-certs\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:57.100865 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.100837 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:57.100865 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.100862 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:57.101013 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.100876 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dskzm for pod openshift-network-diagnostics/network-check-target-mzwbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:57.101013 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.100945 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm podName:e8d3725a-cbd4-4ef3-a433-2010c83a5f3d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:57.600926092 +0000 UTC m=+3.267036260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dskzm" (UniqueName: "kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm") pod "network-check-target-mzwbr" (UID: "e8d3725a-cbd4-4ef3-a433-2010c83a5f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:57.103034 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.103003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqkb\" (UniqueName: \"kubernetes.io/projected/724a681a-c911-439e-aa2c-904081578713-kube-api-access-dsqkb\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.103111 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.103070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/add89f53-e47e-41de-8d0d-6fc06d826d29-kube-api-access-dstxj\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.116510 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.116467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2g542" Nov 03 22:21:57.126910 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.126887 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd4817e_ed9f_4fea_858a_be4d25adcc1f.slice/crio-f1ce9e0ec2670160ecf38165c74868eed7e49647387b153a664244b8a8fd82dd WatchSource:0}: Error finding container f1ce9e0ec2670160ecf38165c74868eed7e49647387b153a664244b8a8fd82dd: Status 404 returned error can't find the container with id f1ce9e0ec2670160ecf38165c74868eed7e49647387b153a664244b8a8fd82dd Nov 03 22:21:57.137462 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.137446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pjjqw" Nov 03 22:21:57.141525 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.141508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:21:57.141569 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.141525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:21:57.143781 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.143759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:21:57.146527 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.146467 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b5b425_585c_4fc9_bc11_2510c7035c70.slice/crio-60c55366c10392824707b262927dd6545f2141c7ca0ee2ae3b98efef3bc1d6fe WatchSource:0}: Error finding container 60c55366c10392824707b262927dd6545f2141c7ca0ee2ae3b98efef3bc1d6fe: Status 404 returned error can't find the container with id 60c55366c10392824707b262927dd6545f2141c7ca0ee2ae3b98efef3bc1d6fe Nov 03 22:21:57.148493 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.148475 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:21:57.152072 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.151951 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f23b7eb_3f0e_4215_a42c_c20c976900dc.slice/crio-6f366bb5ef5917b9fbc4d75946c8250753bac840431368aef2ef57a9768682a5 WatchSource:0}: Error finding container 6f366bb5ef5917b9fbc4d75946c8250753bac840431368aef2ef57a9768682a5: Status 404 returned error can't find the container with id 6f366bb5ef5917b9fbc4d75946c8250753bac840431368aef2ef57a9768682a5 Nov 03 22:21:57.157420 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.157397 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7234dd_7b73_427d_b0e1_afd778bd2028.slice/crio-b81844cd89060dd454ac71b792cc01cfdf446a2e5ff01e9dc4fb444d37946637 WatchSource:0}: Error finding container b81844cd89060dd454ac71b792cc01cfdf446a2e5ff01e9dc4fb444d37946637: Status 404 returned error can't find the container with id b81844cd89060dd454ac71b792cc01cfdf446a2e5ff01e9dc4fb444d37946637 Nov 03 22:21:57.158649 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.158624 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458febd7_7dcd_4ef1_9b8b_27037557b753.slice/crio-60b13fc0432176f1ed91db525e823ea8e00d1c18f237de750d591eebc3058a62 WatchSource:0}: Error finding container 60b13fc0432176f1ed91db525e823ea8e00d1c18f237de750d591eebc3058a62: Status 404 returned error can't find the container with id 60b13fc0432176f1ed91db525e823ea8e00d1c18f237de750d591eebc3058a62 Nov 03 22:21:57.162053 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.162026 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3714e210_a948_4ba6_a174_0fc8eb8e06db.slice/crio-7e8c83649252713af4c4b837e230a4f0b6b86ec21eafcaef4c85bfecee7e22e3 WatchSource:0}: Error finding container 7e8c83649252713af4c4b837e230a4f0b6b86ec21eafcaef4c85bfecee7e22e3: Status 404 returned error can't find the container with id 7e8c83649252713af4c4b837e230a4f0b6b86ec21eafcaef4c85bfecee7e22e3 Nov 03 22:21:57.172763 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.172741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:21:57.174912 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.174896 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:21:57.179149 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.179096 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:21:57.182738 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.182716 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a374ff_7146_47f5_bcbd_6c4dc710d580.slice/crio-e43502be7dc286327ba3930be1ff7900403ba713d6f0d2a5b50b2f2592c69a36 WatchSource:0}: Error finding container e43502be7dc286327ba3930be1ff7900403ba713d6f0d2a5b50b2f2592c69a36: Status 404 returned error can't find the container with id e43502be7dc286327ba3930be1ff7900403ba713d6f0d2a5b50b2f2592c69a36 Nov 03 22:21:57.185891 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.185852 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd89f53_e47e_41de_8d0d_6fc06d826d29.slice/crio-1ee286eb53183ff3d40245d8fbb2be7ee833ac2bf8697cf5d45d12d63789033b WatchSource:0}: Error finding container 1ee286eb53183ff3d40245d8fbb2be7ee833ac2bf8697cf5d45d12d63789033b: Status 404 returned error can't find the container with id 1ee286eb53183ff3d40245d8fbb2be7ee833ac2bf8697cf5d45d12d63789033b Nov 03 22:21:57.188557 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:21:57.188535 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724a681a_c911_439e_aa2c_904081578713.slice/crio-e8bfc9f4803efea7ff26427281671784a21595fd246c625a963545f2036c2508 WatchSource:0}: Error finding container e8bfc9f4803efea7ff26427281671784a21595fd246c625a963545f2036c2508: Status 404 returned error can't find the container with id e8bfc9f4803efea7ff26427281671784a21595fd246c625a963545f2036c2508 Nov 03 22:21:57.440058 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.439973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:57.440220 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.440140 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:57.440220 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.440209 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.440190856 +0000 UTC m=+4.106301023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:57.540948 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.540907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:57.541111 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.541022 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:57.541111 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.541082 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret podName:06b3993b-eaa8-4649-9c38-2b9bb7a22498 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.541068551 +0000 UTC m=+4.207178719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret") pod "global-pull-secret-syncer-c9btg" (UID: "06b3993b-eaa8-4649-9c38-2b9bb7a22498") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:57.642014 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.641963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:21:57.642182 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.642130 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:57.642182 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.642149 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:57.642182 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.642159 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dskzm for pod openshift-network-diagnostics/network-check-target-mzwbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:57.642299 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:57.642212 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm podName:e8d3725a-cbd4-4ef3-a433-2010c83a5f3d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.642197791 +0000 UTC m=+4.308307958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dskzm" (UniqueName: "kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm") pod "network-check-target-mzwbr" (UID: "e8d3725a-cbd4-4ef3-a433-2010c83a5f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:57.794786 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.794707 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"e8bfc9f4803efea7ff26427281671784a21595fd246c625a963545f2036c2508"} Nov 03 22:21:57.796054 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.796028 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" event={"ID":"add89f53-e47e-41de-8d0d-6fc06d826d29","Type":"ContainerStarted","Data":"1ee286eb53183ff3d40245d8fbb2be7ee833ac2bf8697cf5d45d12d63789033b"} Nov 03 22:21:57.797214 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.797168 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvpqf" event={"ID":"c1a374ff-7146-47f5-bcbd-6c4dc710d580","Type":"ContainerStarted","Data":"e43502be7dc286327ba3930be1ff7900403ba713d6f0d2a5b50b2f2592c69a36"} Nov 03 22:21:57.798756 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.798586 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerStarted","Data":"6f366bb5ef5917b9fbc4d75946c8250753bac840431368aef2ef57a9768682a5"} Nov 03 22:21:57.804848 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.804819 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pjjqw" event={"ID":"a9b5b425-585c-4fc9-bc11-2510c7035c70","Type":"ContainerStarted","Data":"60c55366c10392824707b262927dd6545f2141c7ca0ee2ae3b98efef3bc1d6fe"} Nov 03 22:21:57.806104 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.806071 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2g542" event={"ID":"dbd4817e-ed9f-4fea-858a-be4d25adcc1f","Type":"ContainerStarted","Data":"f1ce9e0ec2670160ecf38165c74868eed7e49647387b153a664244b8a8fd82dd"} Nov 03 22:21:57.807385 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.807357 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" event={"ID":"3714e210-a948-4ba6-a174-0fc8eb8e06db","Type":"ContainerStarted","Data":"7e8c83649252713af4c4b837e230a4f0b6b86ec21eafcaef4c85bfecee7e22e3"} Nov 03 22:21:57.809012 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.808990 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qsnsn" event={"ID":"458febd7-7dcd-4ef1-9b8b-27037557b753","Type":"ContainerStarted","Data":"60b13fc0432176f1ed91db525e823ea8e00d1c18f237de750d591eebc3058a62"} Nov 03 22:21:57.810149 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.810120 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fqr6l" event={"ID":"1d7234dd-7b73-427d-b0e1-afd778bd2028","Type":"ContainerStarted","Data":"b81844cd89060dd454ac71b792cc01cfdf446a2e5ff01e9dc4fb444d37946637"} Nov 03 22:21:57.916388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.916304 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:57 +0000 UTC" deadline="2028-08-09 10:34:48.154772016 +0000 UTC" Nov 03 22:21:57.916388 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:57.916375 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15468h12m50.238400761s" Nov 03 22:21:58.449204 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.449159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:58.449393 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.449311 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:58.449393 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.449368 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:00.449351901 +0000 UTC m=+6.115462071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:58.550084 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.550035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:58.550275 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.550193 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:58.550275 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.550253 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret podName:06b3993b-eaa8-4649-9c38-2b9bb7a22498 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:00.550236088 +0000 UTC m=+6.216346256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret") pod "global-pull-secret-syncer-c9btg" (UID: "06b3993b-eaa8-4649-9c38-2b9bb7a22498") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:58.651393 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.651360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:21:58.651653 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.651577 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:58.651653 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.651598 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:58.651653 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.651611 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dskzm for pod openshift-network-diagnostics/network-check-target-mzwbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:58.651851 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.651669 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm podName:e8d3725a-cbd4-4ef3-a433-2010c83a5f3d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:00.651649929 +0000 UTC m=+6.317760101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dskzm" (UniqueName: "kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm") pod "network-check-target-mzwbr" (UID: "e8d3725a-cbd4-4ef3-a433-2010c83a5f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:58.789568 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.789489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:21:58.789731 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.789622 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:21:58.789731 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.789708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:21:58.789890 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.789797 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:21:58.789950 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.789887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:21:58.790012 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:21:58.789980 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:21:58.917071 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.917001 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:57 +0000 UTC" deadline="2028-06-11 03:03:32.832813648 +0000 UTC" Nov 03 22:21:58.917071 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:21:58.917070 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14044h41m33.915750308s" Nov 03 22:22:00.470133 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:00.470093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:00.470836 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.470317 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:00.470836 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.470381 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:04.470360182 +0000 UTC m=+10.136470351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:00.570938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:00.570751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:00.571317 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.571176 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:00.571317 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.571260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret podName:06b3993b-eaa8-4649-9c38-2b9bb7a22498 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:04.571238941 +0000 UTC m=+10.237349098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret") pod "global-pull-secret-syncer-c9btg" (UID: "06b3993b-eaa8-4649-9c38-2b9bb7a22498") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:00.671460 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:00.671082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:00.671460 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.671322 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:00.671460 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.671346 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:00.671460 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.671358 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dskzm for pod openshift-network-diagnostics/network-check-target-mzwbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:00.671460 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.671427 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm podName:e8d3725a-cbd4-4ef3-a433-2010c83a5f3d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:04.671400306 +0000 UTC m=+10.337510477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dskzm" (UniqueName: "kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm") pod "network-check-target-mzwbr" (UID: "e8d3725a-cbd4-4ef3-a433-2010c83a5f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:00.789967 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:00.789430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:00.789967 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.789559 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:00.789967 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:00.789636 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:00.789967 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.789714 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:00.789967 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:00.789778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:00.789967 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:00.789881 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:01.843698 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:01.843400 2578 generic.go:356] "Generic (PLEG): container finished" podID="d4c43c551f0a2ae66c9cba55c176ecea" containerID="e8e30862984064063b791adaa20d337fad7f94bc0ca45085998069f9dfe09895" exitCode=0 Nov 03 22:22:01.844263 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:01.843762 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"d4c43c551f0a2ae66c9cba55c176ecea","Type":"ContainerDied","Data":"e8e30862984064063b791adaa20d337fad7f94bc0ca45085998069f9dfe09895"} Nov 03 22:22:01.852602 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:01.852566 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"3dade66e2d9f1769c6a8531aeeaf63ad","Type":"ContainerStarted","Data":"3a1cbad6f8d0d46f81c985d416b74697a3261ab54595c05ead23a1afa6eea666"} Nov 03 22:22:01.857233 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:01.856726 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerStarted","Data":"c0803a16dab7a8b3f13ff57348fdea10f469c263a365af135256613ee1d8f42e"} Nov 03 22:22:02.104203 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:02.104136 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" podStartSLOduration=5.104112868 podStartE2EDuration="5.104112868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:02.102553511 +0000 UTC m=+7.768663688" watchObservedRunningTime="2026-11-03 22:22:02.104112868 +0000 UTC m=+7.770223043" Nov 03 22:22:02.789632 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:02.789367 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:02.789632 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:02.789515 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:02.789966 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:02.789945 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:02.790091 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:02.790057 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:02.790208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:02.790129 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:02.790270 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:02.790205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:02.870091 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:02.870017 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="c0803a16dab7a8b3f13ff57348fdea10f469c263a365af135256613ee1d8f42e" exitCode=0 Nov 03 22:22:02.870553 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:02.870109 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerDied","Data":"c0803a16dab7a8b3f13ff57348fdea10f469c263a365af135256613ee1d8f42e"} Nov 03 22:22:02.880516 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:02.880454 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"d4c43c551f0a2ae66c9cba55c176ecea","Type":"ContainerStarted","Data":"edbf084cb9d3618769e229e3c3e6925e17d0a5cc0851136c5f896577b47f6ee5"} Nov 03 22:22:03.140643 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:03.140528 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" podStartSLOduration=6.140505632 podStartE2EDuration="6.140505632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:03.140081552 +0000 UTC m=+8.806191727" watchObservedRunningTime="2026-11-03 22:22:03.140505632 +0000 UTC m=+8.806615811" Nov 03 22:22:03.887357 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:03.887314 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvpqf" event={"ID":"c1a374ff-7146-47f5-bcbd-6c4dc710d580","Type":"ContainerStarted","Data":"34b2535434d4a2e00dd872075406f30d1b96fc51f8224525aa0126ad69f547ac"} Nov 03 22:22:03.891816 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:03.891787 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2g542" event={"ID":"dbd4817e-ed9f-4fea-858a-be4d25adcc1f","Type":"ContainerStarted","Data":"b65e985c6f72bce9235fe567fadedc1da5cc16a3aa8b0898693429a158257453"} Nov 03 22:22:03.932743 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:03.932666 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:04.014459 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.014392 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pvpqf" podStartSLOduration=2.575343261 podStartE2EDuration="8.014371111s" totalImagesPullingTime="5.43902785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:04.013171402 +0000 UTC m=+9.679281579" watchObservedRunningTime="2026-11-03 22:22:04.014371111 +0000 UTC m=+9.680481288" Nov 03 22:22:04.132704 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.132652 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2g542" podStartSLOduration=2.608985543 podStartE2EDuration="9.132631129s" totalImagesPullingTime="6.523645586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:04.132128983 +0000 UTC m=+9.798239156" watchObservedRunningTime="2026-11-03 22:22:04.132631129 +0000 UTC m=+9.798741304" Nov 03 22:22:04.503892 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.503864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:04.504075 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.504059 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:04.504143 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.504112 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:12.504099649 +0000 UTC m=+18.170209813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:04.605415 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.604948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:04.605415 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.605063 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:04.605415 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.605110 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret podName:06b3993b-eaa8-4649-9c38-2b9bb7a22498 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:12.605098262 +0000 UTC m=+18.271208411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret") pod "global-pull-secret-syncer-c9btg" (UID: "06b3993b-eaa8-4649-9c38-2b9bb7a22498") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:04.705802 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.705751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:04.706029 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.706010 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:04.706103 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.706036 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:04.706103 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.706055 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dskzm for pod openshift-network-diagnostics/network-check-target-mzwbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:04.706190 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.706120 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm podName:e8d3725a-cbd4-4ef3-a433-2010c83a5f3d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:12.706101892 +0000 UTC m=+18.372212047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dskzm" (UniqueName: "kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm") pod "network-check-target-mzwbr" (UID: "e8d3725a-cbd4-4ef3-a433-2010c83a5f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:04.788897 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.788866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:04.789077 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.788986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:04.789077 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.789066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:04.789207 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.789139 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:04.789207 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.789197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:04.789311 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:04.789279 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:04.898380 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:04.897913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" event={"ID":"add89f53-e47e-41de-8d0d-6fc06d826d29","Type":"ContainerStarted","Data":"7068e66af99208361fca896b2a0e2a75ee029373d86e81fb16d3401c76bf2fd3"} Nov 03 22:22:06.441458 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.441404 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:22:06.786279 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.786240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:06.786279 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.786255 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:06.786279 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.786289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:06.786522 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:06.786411 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:06.786565 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:06.786515 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:06.786618 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:06.786600 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:06.886969 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.886850 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:22:06.441424378Z","UUID":"bd52d71f-9dd6-4752-838f-9084e4a17f35","Handler":null,"Name":"","Endpoint":""} Nov 03 22:22:06.888587 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.888566 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:22:06.888716 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.888597 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:22:06.904757 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.904141 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fqr6l" event={"ID":"1d7234dd-7b73-427d-b0e1-afd778bd2028","Type":"ContainerStarted","Data":"efa2a5ee44c65d61fc135e22f2f5fca819573d9cf828c54cf92114df8ad5f8d8"} Nov 03 22:22:06.906896 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.906842 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" event={"ID":"add89f53-e47e-41de-8d0d-6fc06d826d29","Type":"ContainerStarted","Data":"c36633f9800f8f3b2797ae2d637a4666218d7a03f8b3c7f717038d398d9f08be"} Nov 03 22:22:06.908560 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:06.908539 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pjjqw" event={"ID":"a9b5b425-585c-4fc9-bc11-2510c7035c70","Type":"ContainerStarted","Data":"e1521cdc7d2607887eba3e485db6862394c3a6d76fc5bed97f88037510b9ab27"} Nov 03 22:22:07.057994 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:07.057694 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-fqr6l" podStartSLOduration=2.9253835280000002 podStartE2EDuration="12.057675978s" totalImagesPullingTime="9.13229245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:07.057354069 +0000 UTC m=+12.723464243" watchObservedRunningTime="2026-11-03 22:22:07.057675978 +0000 UTC m=+12.723786152" Nov 03 22:22:07.913817 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:07.913419 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="c8fee35c220045e2c029953e76442373b6be086736fc41d937ba045c205715d2" exitCode=0 Nov 03 22:22:07.914731 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:07.914508 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerDied","Data":"c8fee35c220045e2c029953e76442373b6be086736fc41d937ba045c205715d2"} Nov 03 22:22:08.137499 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.137449 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pjjqw" podStartSLOduration=3.916180705 podStartE2EDuration="13.137433807s" totalImagesPullingTime="9.221253102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:07.178503492 +0000 UTC m=+12.844613664" watchObservedRunningTime="2026-11-03 22:22:08.137433807 +0000 UTC m=+13.803543980" Nov 03 22:22:08.595951 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.595919 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:08.596493 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.596472 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:08.786043 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.785973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:08.786178 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.785983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:08.786178 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.786098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:08.786178 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:08.786103 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:08.786274 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:08.786226 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:08.786349 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:08.786329 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:08.918460 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.918424 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" event={"ID":"add89f53-e47e-41de-8d0d-6fc06d826d29","Type":"ContainerStarted","Data":"5c08e3492459a9e43fe392df6908c8637e00071266f0a2fb97219448979d87b4"} Nov 03 22:22:08.920227 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.920195 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" event={"ID":"3714e210-a948-4ba6-a174-0fc8eb8e06db","Type":"ContainerStarted","Data":"a86994b0cbc3fb82282af8325444a1b84bb79d2591271e9be16999ff1157883d"} Nov 03 22:22:08.921855 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.921825 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qsnsn" event={"ID":"458febd7-7dcd-4ef1-9b8b-27037557b753","Type":"ContainerStarted","Data":"d302d24183572f125936043965e80ffa020e957ede135c30da1b494ffa9494ed"} Nov 03 22:22:08.922514 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:08.922493 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:09.040832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:09.040716 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" podStartSLOduration=2.555237693 podStartE2EDuration="13.040695562s" totalImagesPullingTime="10.485457869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:09.040167331 +0000 UTC m=+14.706277504" watchObservedRunningTime="2026-11-03 22:22:09.040695562 +0000 UTC m=+14.706805736" Nov 03 22:22:09.157338 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:09.157286 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qsnsn" podStartSLOduration=4.78991462 podStartE2EDuration="14.157267277s" totalImagesPullingTime="9.367352657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:09.156208999 +0000 UTC m=+14.822319173" watchObservedRunningTime="2026-11-03 22:22:09.157267277 +0000 UTC m=+14.823377450" Nov 03 22:22:09.275016 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:09.274960 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" podStartSLOduration=3.088270184 podStartE2EDuration="14.274944917s" totalImagesPullingTime="11.186674733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:09.274466868 +0000 UTC m=+14.940577039" watchObservedRunningTime="2026-11-03 22:22:09.274944917 +0000 UTC m=+14.941055090" Nov 03 22:22:09.925860 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:09.925826 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="5f8afe677d7e40921f2ac1af605b4db4c7db1369bd9c9eefef4514c4c1919ddf" exitCode=0 Nov 03 22:22:09.926455 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:09.925923 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerDied","Data":"5f8afe677d7e40921f2ac1af605b4db4c7db1369bd9c9eefef4514c4c1919ddf"} Nov 03 22:22:10.787096 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:10.787059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:10.787096 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:10.787087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:10.787313 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:10.787176 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:10.787313 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:10.787234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:10.787435 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:10.787342 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:10.787481 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:10.787432 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:11.933621 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:11.933360 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="c9ab69928c2b0652d7a9a2069c8a1e5c3462a1308f37f5946dc6a4acfc15efb4" exitCode=0 Nov 03 22:22:11.934038 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:11.933438 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerDied","Data":"c9ab69928c2b0652d7a9a2069c8a1e5c3462a1308f37f5946dc6a4acfc15efb4"} Nov 03 22:22:12.567558 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:12.567526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:12.567729 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.567638 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:12.567729 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.567689 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.567674344 +0000 UTC m=+34.233784508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:12.668694 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:12.668661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:12.668844 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.668762 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:12.668844 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.668826 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret podName:06b3993b-eaa8-4649-9c38-2b9bb7a22498 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.668812469 +0000 UTC m=+34.334922619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret") pod "global-pull-secret-syncer-c9btg" (UID: "06b3993b-eaa8-4649-9c38-2b9bb7a22498") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:12.769743 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:12.769710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:12.769893 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.769877 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:12.769953 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.769894 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:12.769953 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.769905 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dskzm for pod openshift-network-diagnostics/network-check-target-mzwbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:12.769953 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.769951 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm podName:e8d3725a-cbd4-4ef3-a433-2010c83a5f3d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.769940072 +0000 UTC m=+34.436050221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dskzm" (UniqueName: "kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm") pod "network-check-target-mzwbr" (UID: "e8d3725a-cbd4-4ef3-a433-2010c83a5f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:12.786938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:12.786911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:12.786938 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:12.786938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:12.787123 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:12.786950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:12.787123 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.787051 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:12.787285 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.787146 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:12.787285 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:12.787219 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:12.937577 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:12.937547 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"28a8b9cd222021df412f566ae555597af6306db4da7efa491f4d79f8dc3d992a"} Nov 03 22:22:13.943934 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:13.943699 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:22:13.944419 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:13.944302 2578 generic.go:356] "Generic (PLEG): container finished" podID="724a681a-c911-439e-aa2c-904081578713" containerID="ffc553080bd220a81fafed5b87ebe6866d10f4aaf755d191e9f6c9e0e6f26216" exitCode=1 Nov 03 22:22:13.944419 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:13.944363 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"51ecab35ba5330c6f0d215d0324e71920e9dc378bdf9080beee4f8abe23c7b8a"} Nov 03 22:22:13.946693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:13.944715 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"a5c3be283ff844f7ff3260ba0a39eec6cb221732f1d38c7db81c2b5a0e75c2b6"} Nov 03 22:22:13.946693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:13.944891 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"0fbcb453a0a13583180fe3c028458f79675a31d4ab0d06994009c39ebdebd819"} Nov 03 22:22:13.946693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:13.944990 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"ed557f64c6ef4e9aee7e37f3d93751af33c18d8b7f54bfea1ec73656549d6ae1"} Nov 03 22:22:13.946693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:13.945057 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerDied","Data":"ffc553080bd220a81fafed5b87ebe6866d10f4aaf755d191e9f6c9e0e6f26216"} Nov 03 22:22:14.787039 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:14.787004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:14.787386 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:14.787128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:14.787386 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:14.787140 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:14.787386 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:14.787224 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:14.787386 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:14.787272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:14.787386 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:14.787347 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:14.948973 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:14.948940 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="bdbbf881f4defa5095fa920522bccad0451a764d7724b11b19cdf4fddccad9ce" exitCode=0 Nov 03 22:22:14.949479 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:14.948983 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerDied","Data":"bdbbf881f4defa5095fa920522bccad0451a764d7724b11b19cdf4fddccad9ce"} Nov 03 22:22:15.953752 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:15.953718 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="7375d0346f8aa57c4088ec3f28b66e7bd849041ce33efc9773b4267e61970205" exitCode=0 Nov 03 22:22:15.954157 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:15.953812 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerDied","Data":"7375d0346f8aa57c4088ec3f28b66e7bd849041ce33efc9773b4267e61970205"} Nov 03 22:22:15.956975 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:15.956957 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:22:15.957335 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:15.957317 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"39f7f458edc8870e349069e69c9271af9eae1618361a5392f0dcab39fbfef987"} Nov 03 22:22:16.786206 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:16.786173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:16.786206 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:16.786206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:16.786396 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:16.786275 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:16.786396 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:16.786332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:16.786517 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:16.786401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:16.786517 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:16.786483 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:16.962563 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:16.962531 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqclr" event={"ID":"1f23b7eb-3f0e-4215-a42c-c20c976900dc","Type":"ContainerStarted","Data":"1d068fc57ba028169189a0cf216ef15b0a464d25844bff6d6815f495a294a0b7"} Nov 03 22:22:17.085552 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:17.085503 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jqclr" podStartSLOduration=9.435473099 podStartE2EDuration="22.085490243s" totalImagesPullingTime="12.650017144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:17.08495748 +0000 UTC m=+22.751067652" watchObservedRunningTime="2026-11-03 22:22:17.085490243 +0000 UTC m=+22.751600434" Nov 03 22:22:17.967881 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:17.967683 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:22:17.968333 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:17.968269 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"528e1766299cd486a8d13ffca061d143c4b46aedac8d38168dcc5016ef2328cb"} Nov 03 22:22:17.968824 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:17.968805 2578 scope.go:122] "RemoveContainer" containerID="ffc553080bd220a81fafed5b87ebe6866d10f4aaf755d191e9f6c9e0e6f26216" Nov 03 22:22:18.789010 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.788974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:18.789450 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:18.789422 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:18.790000 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.789980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:18.790237 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:18.790218 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:18.790568 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.790552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:18.790767 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:18.790749 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:18.973353 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.973325 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:22:18.973747 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.973714 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" event={"ID":"724a681a-c911-439e-aa2c-904081578713","Type":"ContainerStarted","Data":"9732d2a0d1d79913bd6624ccf4731a9df463871c666c08391606d10af29357de"} Nov 03 22:22:18.973997 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.973975 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:18.974127 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.974063 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:18.974127 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.974112 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:18.988474 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.988451 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:18.988568 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:18.988546 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:19.183299 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:19.183230 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" podStartSLOduration=8.561546868 podStartE2EDuration="24.183215107s" totalImagesPullingTime="15.621668239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:19.15484425 +0000 UTC m=+24.820954422" watchObservedRunningTime="2026-11-03 22:22:19.183215107 +0000 UTC m=+24.849325317" Nov 03 22:22:20.271218 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:20.271181 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c9btg"] Nov 03 22:22:20.271706 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:20.271295 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:20.271706 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:20.271402 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:20.274272 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:20.274246 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mzwbr"] Nov 03 22:22:20.274382 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:20.274337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:20.274454 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:20.274433 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:20.276862 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:20.276845 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rcrd8"] Nov 03 22:22:20.276937 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:20.276926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:20.277021 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:20.277006 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:21.786278 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:21.786234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:21.786278 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:21.786264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:21.786278 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:21.786287 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:21.786692 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:21.786377 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:21.786692 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:21.786502 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:21.786692 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:21.786583 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:23.786855 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:23.786819 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:23.787294 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:23.786823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:23.787294 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:23.786923 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:23.787294 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:23.787039 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:23.787294 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:23.786823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:23.787294 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:23.787130 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:25.786622 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:25.786445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:25.787061 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:25.786476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:25.787061 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:25.786685 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9btg" podUID="06b3993b-eaa8-4649-9c38-2b9bb7a22498" Nov 03 22:22:25.787061 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:25.786526 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:25.787061 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:25.786837 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcrd8" podUID="de375da1-1b02-4796-ad49-2062bc5bc74b" Nov 03 22:22:25.787061 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:25.786862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mzwbr" podUID="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" Nov 03 22:22:26.205808 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.205767 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeReady" Nov 03 22:22:26.205956 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.205924 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:22:26.322368 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.322342 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j"] Nov 03 22:22:26.336641 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.336615 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xrnfv"] Nov 03 22:22:26.336771 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.336752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" Nov 03 22:22:26.350676 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.350658 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6ddbh"] Nov 03 22:22:26.350802 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.350769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" Nov 03 22:22:26.364701 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.364682 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hdlkm"] Nov 03 22:22:26.364818 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.364772 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" Nov 03 22:22:26.378962 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.378943 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d9b64b6dc-xqdzm"] Nov 03 22:22:26.379076 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.379060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" Nov 03 22:22:26.391096 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.391080 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.391369 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.391354 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.391481 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.391464 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gg966\"" Nov 03 22:22:26.401277 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.401255 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg"] Nov 03 22:22:26.401482 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.401455 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.429400 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.429356 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jkxgz"] Nov 03 22:22:26.429526 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.429480 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.445627 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.445603 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6"] Nov 03 22:22:26.445765 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.445733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:26.465838 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.465816 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z5lhd"] Nov 03 22:22:26.465985 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.465967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.468846 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.468777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwt5\" (UniqueName: \"kubernetes.io/projected/e3939775-883b-454d-9e3b-6949f6984dfb-kube-api-access-5gwt5\") pod \"kas-connection-checker-685895c7c9-6ddbh\" (UID: \"e3939775-883b-454d-9e3b-6949f6984dfb\") " pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" Nov 03 22:22:26.468954 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.468893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qrs\" (UniqueName: \"kubernetes.io/projected/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34-kube-api-access-j9qrs\") pod \"network-check-source-6f78d59f99-7hx9j\" (UID: \"3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" Nov 03 22:22:26.469023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.468964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqnx\" (UniqueName: \"kubernetes.io/projected/35ea235a-c0ba-44b9-909f-e3dd472a6c36-kube-api-access-cgqnx\") pod \"kas-connection-checker-685895c7c9-hdlkm\" (UID: \"35ea235a-c0ba-44b9-909f-e3dd472a6c36\") " pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" Nov 03 22:22:26.469023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.468998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqj6s\" (UniqueName: \"kubernetes.io/projected/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff-kube-api-access-hqj6s\") pod \"kas-connection-checker-685895c7c9-xrnfv\" (UID: \"e0a3e17d-abf7-4d4b-afe0-fd028d2caeff\") " pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" Nov 03 22:22:26.479982 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.479961 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58d49cb494-tnd66"] Nov 03 22:22:26.480084 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.480069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.491272 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.491254 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:22:26.491538 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.491520 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:22:26.491612 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.491564 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5wr4r\"" Nov 03 22:22:26.491612 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.491604 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bwztx\"" Nov 03 22:22:26.491813 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.491782 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:22:26.491888 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.491875 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:22:26.493050 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.493035 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.493050 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.493046 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.493163 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.493061 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.493163 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.493063 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.493163 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.493116 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:22:26.504007 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.503982 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74d647c79-nsnqv"] Nov 03 22:22:26.504109 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.504094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.518188 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.518165 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs"] Nov 03 22:22:26.518274 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.518230 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:22:26.518330 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.518288 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zvxhn\"" Nov 03 22:22:26.518386 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.518353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.519090 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.519069 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:22:26.527675 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.527660 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:22:26.532719 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.532702 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw"] Nov 03 22:22:26.532842 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.532828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" Nov 03 22:22:26.552595 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.552571 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5"] Nov 03 22:22:26.552704 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.552690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.570242 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qrs\" (UniqueName: \"kubernetes.io/projected/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34-kube-api-access-j9qrs\") pod \"network-check-source-6f78d59f99-7hx9j\" (UID: \"3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" Nov 03 22:22:26.570352 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-tmp\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.570352 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-snapshots\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.570352 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92f9\" (UniqueName: \"kubernetes.io/projected/d5210f23-6d88-413f-b0b5-c1b938713ca9-kube-api-access-c92f9\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.570352 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.570537 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpk4w\" (UniqueName: \"kubernetes.io/projected/60be4f8c-7051-444d-ad87-7e7895c94542-kube-api-access-hpk4w\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.570537 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.570537 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-service-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.570537 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.570537 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.570665 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.570665 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:26.570665 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.570665 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.570847 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqnx\" (UniqueName: \"kubernetes.io/projected/35ea235a-c0ba-44b9-909f-e3dd472a6c36-kube-api-access-cgqnx\") pod \"kas-connection-checker-685895c7c9-hdlkm\" (UID: \"35ea235a-c0ba-44b9-909f-e3dd472a6c36\") " pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" Nov 03 22:22:26.570847 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqj6s\" (UniqueName: \"kubernetes.io/projected/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff-kube-api-access-hqj6s\") pod \"kas-connection-checker-685895c7c9-xrnfv\" (UID: \"e0a3e17d-abf7-4d4b-afe0-fd028d2caeff\") " pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" Nov 03 22:22:26.570847 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.570991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.570991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5210f23-6d88-413f-b0b5-c1b938713ca9-serving-cert\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.570991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60be4f8c-7051-444d-ad87-7e7895c94542-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.570991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.570991 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.571150 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.570993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.571150 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.571093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwt5\" (UniqueName: \"kubernetes.io/projected/e3939775-883b-454d-9e3b-6949f6984dfb-kube-api-access-5gwt5\") pod \"kas-connection-checker-685895c7c9-6ddbh\" (UID: \"e3939775-883b-454d-9e3b-6949f6984dfb\") " pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" Nov 03 22:22:26.571150 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.571117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnfb\" (UniqueName: \"kubernetes.io/projected/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-kube-api-access-kpnfb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.571238 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.571210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f621d2a-1926-4044-80bb-d64a0574e356-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:26.574693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.574658 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-thjpw"] Nov 03 22:22:26.574829 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.574815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.590724 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.590703 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:22:26.590898 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.590881 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:22:26.590972 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.590947 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:22:26.590972 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.590970 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-khx6v\"" Nov 03 22:22:26.591134 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.591028 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8l7wg\"" Nov 03 22:22:26.591134 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.591040 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:22:26.591134 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.591084 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:22:26.591134 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.591093 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x6gxc\"" Nov 03 22:22:26.591258 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.591155 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-t48dr\"" Nov 03 22:22:26.593130 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593117 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.593179 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593143 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.593236 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593193 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:22:26.593236 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593226 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:22:26.593305 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593226 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.593305 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593257 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:22:26.593305 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593241 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.593305 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593240 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.593305 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593284 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.593305 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593241 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.593513 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.593338 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.594297 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.594278 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:22:26.598348 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.598332 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jmjbp"] Nov 03 22:22:26.598447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.598436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.612546 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.612525 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pqv9b"] Nov 03 22:22:26.612674 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.612657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.630835 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.630816 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j"] Nov 03 22:22:26.630936 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.630885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:26.630936 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.630907 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xrnfv"] Nov 03 22:22:26.630936 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.630936 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d9b64b6dc-xqdzm"] Nov 03 22:22:26.631093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.630961 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6"] Nov 03 22:22:26.631093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.630986 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg"] Nov 03 22:22:26.631093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631006 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z5lhd"] Nov 03 22:22:26.631093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631028 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d49cb494-tnd66"] Nov 03 22:22:26.631093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631049 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-thjpw"] Nov 03 22:22:26.631093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631068 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jmjbp"] Nov 03 22:22:26.631093 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631094 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw"] Nov 03 22:22:26.631390 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631120 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6ddbh"] Nov 03 22:22:26.631390 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631139 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74d647c79-nsnqv"] Nov 03 22:22:26.631390 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631158 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hdlkm"] Nov 03 22:22:26.631390 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631183 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jkxgz"] Nov 03 22:22:26.631390 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631204 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs"] Nov 03 22:22:26.631390 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631224 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5"] Nov 03 22:22:26.631390 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.631245 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqv9b"] Nov 03 22:22:26.633886 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.633866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqnx\" (UniqueName: \"kubernetes.io/projected/35ea235a-c0ba-44b9-909f-e3dd472a6c36-kube-api-access-cgqnx\") pod \"kas-connection-checker-685895c7c9-hdlkm\" (UID: \"35ea235a-c0ba-44b9-909f-e3dd472a6c36\") " pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" Nov 03 22:22:26.633975 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.633869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqj6s\" (UniqueName: \"kubernetes.io/projected/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff-kube-api-access-hqj6s\") pod \"kas-connection-checker-685895c7c9-xrnfv\" (UID: \"e0a3e17d-abf7-4d4b-afe0-fd028d2caeff\") " pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" Nov 03 22:22:26.634067 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.634048 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwt5\" (UniqueName: \"kubernetes.io/projected/e3939775-883b-454d-9e3b-6949f6984dfb-kube-api-access-5gwt5\") pod \"kas-connection-checker-685895c7c9-6ddbh\" (UID: \"e3939775-883b-454d-9e3b-6949f6984dfb\") " pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" Nov 03 22:22:26.634133 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.634120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qrs\" (UniqueName: \"kubernetes.io/projected/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34-kube-api-access-j9qrs\") pod \"network-check-source-6f78d59f99-7hx9j\" (UID: \"3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" Nov 03 22:22:26.659910 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.659886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" Nov 03 22:22:26.671488 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.671470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" Nov 03 22:22:26.671707 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.671688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60be4f8c-7051-444d-ad87-7e7895c94542-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.671811 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.671727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l29j\" (UniqueName: \"kubernetes.io/projected/65f4d5dd-a512-425a-91b9-de7749f091c6-kube-api-access-4l29j\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.671811 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.671758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fh74\" (UniqueName: \"kubernetes.io/projected/b6fe9b11-1b54-4871-9236-dae9154b43fb-kube-api-access-4fh74\") pod \"volume-data-source-validator-58c9f5f46d-zwxcs\" (UID: \"b6fe9b11-1b54-4871-9236-dae9154b43fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" Nov 03 22:22:26.671811 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.671801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghp7z\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.671970 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.671870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.671970 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.671929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.672153 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.672218 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a1808a-9488-4d49-8028-4860660c0c9d-serving-cert\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.672276 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-stats-auth\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.672342 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnfb\" (UniqueName: \"kubernetes.io/projected/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-kube-api-access-kpnfb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.672400 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672351 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60be4f8c-7051-444d-ad87-7e7895c94542-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.672400 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xjb\" (UniqueName: \"kubernetes.io/projected/64ebd614-43bd-479e-8e95-c33b2c93361c-kube-api-access-g6xjb\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.672493 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f621d2a-1926-4044-80bb-d64a0574e356-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:26.672493 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-tmp\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.672493 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk4w\" (UniqueName: \"kubernetes.io/projected/60be4f8c-7051-444d-ad87-7e7895c94542-kube-api-access-hpk4w\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.672645 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.672645 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65f4d5dd-a512-425a-91b9-de7749f091c6-serving-cert\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.672645 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-snapshots\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.672645 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92f9\" (UniqueName: \"kubernetes.io/projected/d5210f23-6d88-413f-b0b5-c1b938713ca9-kube-api-access-c92f9\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.672645 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.672645 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pmx\" (UniqueName: \"kubernetes.io/projected/700b9d15-342a-46d1-b94e-6b44aae2d9b5-kube-api-access-g5pmx\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-tmp\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5210f23-6d88-413f-b0b5-c1b938713ca9-serving-cert\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.672957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-config\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.673365 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.672967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.673365 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.673007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.673365 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.673043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f621d2a-1926-4044-80bb-d64a0574e356-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:26.673365 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.673047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f4d5dd-a512-425a-91b9-de7749f091c6-config\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.673365 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.673086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.673365 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.673165 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:26.673365 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.673218 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.173199334 +0000 UTC m=+32.839309516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:26.673700 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.673412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-snapshots\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.673700 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.673680 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.674000 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.673980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.674110 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.674294 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.674275 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:26.674294 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.674294 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:26.674453 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674400 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.674654 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.674746 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.674833 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-service-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.674833 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.674945 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.674945 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.674926 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.174911104 +0000 UTC m=+32.841021273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:26.675058 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.674982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.675058 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fkn\" (UniqueName: \"kubernetes.io/projected/e3a1808a-9488-4d49-8028-4860660c0c9d-kube-api-access-79fkn\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.675158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.675158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.675158 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.675315 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:26.675315 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.675315 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-service-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.675315 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.675315 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.675540 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.675342 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:26.675540 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-default-certificate\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.675540 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.675393 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.175374264 +0000 UTC m=+32.841484429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:26.675540 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.675393 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5210f23-6d88-413f-b0b5-c1b938713ca9-serving-cert\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.676441 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.676416 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.677320 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.677301 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.691429 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.691411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" Nov 03 22:22:26.691899 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.691824 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:22:26.691899 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.691846 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:22:26.692014 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.691874 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:22:26.692014 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.691889 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:22:26.692175 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.692160 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:22:26.692240 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.692203 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v9lfz\"" Nov 03 22:22:26.692240 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.692200 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h9m4r\"" Nov 03 22:22:26.692306 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.692247 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dd97f\"" Nov 03 22:22:26.692306 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.692272 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2jqrs\"" Nov 03 22:22:26.692370 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.692343 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pq9zt\"" Nov 03 22:22:26.693021 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693001 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.693238 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693043 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:22:26.693238 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693053 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:22:26.693238 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693068 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.693238 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693192 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.693238 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693209 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.693832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693247 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.693832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693261 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:22:26.693832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693268 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:22:26.693832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693306 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.693832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693317 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:26.693832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.693340 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:26.694862 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.694843 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:22:26.712205 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.712187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" Nov 03 22:22:26.737958 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.737906 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnfb\" (UniqueName: \"kubernetes.io/projected/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-kube-api-access-kpnfb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.740445 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.739013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.740445 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.739300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92f9\" (UniqueName: \"kubernetes.io/projected/d5210f23-6d88-413f-b0b5-c1b938713ca9-kube-api-access-c92f9\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.741212 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.741169 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:26.741858 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.741826 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk4w\" (UniqueName: \"kubernetes.io/projected/60be4f8c-7051-444d-ad87-7e7895c94542-kube-api-access-hpk4w\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:26.777306 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.777206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/7c95c406-8766-4a77-9639-513f2f37c1d3-kube-api-access-sbfgm\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:26.777507 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.777482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e1c673-629b-4840-9f7a-aa64777cf32a-config-volume\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.777746 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.777724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.778062 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.778034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.778379 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.777975 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:26.778379 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.778286 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:26.778379 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.778351 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.278329752 +0000 UTC m=+32.944439903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:26.779305 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.779256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f4d5dd-a512-425a-91b9-de7749f091c6-config\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.779525 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.779411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f4d5dd-a512-425a-91b9-de7749f091c6-config\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.779525 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.779471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.780256 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.780227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.780342 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.780315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.780656 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.780343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jpx\" (UniqueName: \"kubernetes.io/projected/68e1c673-629b-4840-9f7a-aa64777cf32a-kube-api-access-j5jpx\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.780763 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.780742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.780931 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.780866 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:26.780931 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.780915 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.280899717 +0000 UTC m=+32.947009890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:26.781034 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.780943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c95c406-8766-4a77-9639-513f2f37c1d3-cert\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:26.781034 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.780979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fkn\" (UniqueName: \"kubernetes.io/projected/e3a1808a-9488-4d49-8028-4860660c0c9d-kube-api-access-79fkn\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.781124 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.781047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.781124 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.781099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.781222 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.781167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.781222 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.781190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e1c673-629b-4840-9f7a-aa64777cf32a-tmp-dir\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.781524 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.781501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.783533 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.783533 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.783533 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.783533 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-default-certificate\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l29j\" (UniqueName: \"kubernetes.io/projected/65f4d5dd-a512-425a-91b9-de7749f091c6-kube-api-access-4l29j\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fh74\" (UniqueName: \"kubernetes.io/projected/b6fe9b11-1b54-4871-9236-dae9154b43fb-kube-api-access-4fh74\") pod \"volume-data-source-validator-58c9f5f46d-zwxcs\" (UID: \"b6fe9b11-1b54-4871-9236-dae9154b43fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghp7z\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a1808a-9488-4d49-8028-4860660c0c9d-serving-cert\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-stats-auth\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xjb\" (UniqueName: \"kubernetes.io/projected/64ebd614-43bd-479e-8e95-c33b2c93361c-kube-api-access-g6xjb\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65f4d5dd-a512-425a-91b9-de7749f091c6-serving-cert\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.784023 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.783856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pmx\" (UniqueName: \"kubernetes.io/projected/700b9d15-342a-46d1-b94e-6b44aae2d9b5-kube-api-access-g5pmx\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.785993 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.786060 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.286042022 +0000 UTC m=+32.952152190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.786242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.786307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-config\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.787039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-config\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.787271 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.287249899 +0000 UTC m=+32.953360069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.792137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a1808a-9488-4d49-8028-4860660c0c9d-serving-cert\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.792677 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.792639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.794509 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.794046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:26.800616 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.800591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-default-certificate\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.801338 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.800783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-stats-auth\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.801338 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.801318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.801931 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.801914 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65f4d5dd-a512-425a-91b9-de7749f091c6-serving-cert\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.803302 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.803139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.815164 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.814438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:26.844347 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.844320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.844845 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.844823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghp7z\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:26.845611 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.845585 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fkn\" (UniqueName: \"kubernetes.io/projected/e3a1808a-9488-4d49-8028-4860660c0c9d-kube-api-access-79fkn\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.848598 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.848576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l29j\" (UniqueName: \"kubernetes.io/projected/65f4d5dd-a512-425a-91b9-de7749f091c6-kube-api-access-4l29j\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.852007 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.851422 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pmx\" (UniqueName: \"kubernetes.io/projected/700b9d15-342a-46d1-b94e-6b44aae2d9b5-kube-api-access-g5pmx\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:26.852007 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.851965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fh74\" (UniqueName: \"kubernetes.io/projected/b6fe9b11-1b54-4871-9236-dae9154b43fb-kube-api-access-4fh74\") pod \"volume-data-source-validator-58c9f5f46d-zwxcs\" (UID: \"b6fe9b11-1b54-4871-9236-dae9154b43fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" Nov 03 22:22:26.852928 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.852905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xjb\" (UniqueName: \"kubernetes.io/projected/64ebd614-43bd-479e-8e95-c33b2c93361c-kube-api-access-g6xjb\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:26.875354 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.875327 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" Nov 03 22:22:26.887853 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.887772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/7c95c406-8766-4a77-9639-513f2f37c1d3-kube-api-access-sbfgm\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:26.887853 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.887824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e1c673-629b-4840-9f7a-aa64777cf32a-config-volume\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.887957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.887867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.887957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.887913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jpx\" (UniqueName: \"kubernetes.io/projected/68e1c673-629b-4840-9f7a-aa64777cf32a-kube-api-access-j5jpx\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.887957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.887950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c95c406-8766-4a77-9639-513f2f37c1d3-cert\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:26.888051 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.888001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e1c673-629b-4840-9f7a-aa64777cf32a-tmp-dir\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.888432 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.888406 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:26.888894 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:26.888485 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:27.388464108 +0000 UTC m=+33.054574273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:26.889044 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.889023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e1c673-629b-4840-9f7a-aa64777cf32a-tmp-dir\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.889184 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.889165 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e1c673-629b-4840-9f7a-aa64777cf32a-config-volume\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.892061 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.892009 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c95c406-8766-4a77-9639-513f2f37c1d3-cert\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:26.935743 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.935714 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:26.952109 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.952076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:26.958434 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.958405 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/7c95c406-8766-4a77-9639-513f2f37c1d3-kube-api-access-sbfgm\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:26.958530 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.958517 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jpx\" (UniqueName: \"kubernetes.io/projected/68e1c673-629b-4840-9f7a-aa64777cf32a-kube-api-access-j5jpx\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:26.967205 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:26.967183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:27.096660 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.096577 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xrnfv"] Nov 03 22:22:27.097891 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.097846 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j"] Nov 03 22:22:27.098377 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.098351 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6ddbh"] Nov 03 22:22:27.100164 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.100141 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc089dd_eb07_47b1_a9bf_e1bc4e44aa34.slice/crio-23bfb68550386e57bbfef9fc5811d99c33613e35a85be46da473af252806b87c WatchSource:0}: Error finding container 23bfb68550386e57bbfef9fc5811d99c33613e35a85be46da473af252806b87c: Status 404 returned error can't find the container with id 23bfb68550386e57bbfef9fc5811d99c33613e35a85be46da473af252806b87c Nov 03 22:22:27.100800 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.100761 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a3e17d_abf7_4d4b_afe0_fd028d2caeff.slice/crio-478ac8d747bd642b9bfb5ed1136dbc4c3b8ed747122f7dbeb5923b0d57f05ac0 WatchSource:0}: Error finding container 478ac8d747bd642b9bfb5ed1136dbc4c3b8ed747122f7dbeb5923b0d57f05ac0: Status 404 returned error can't find the container with id 478ac8d747bd642b9bfb5ed1136dbc4c3b8ed747122f7dbeb5923b0d57f05ac0 Nov 03 22:22:27.101514 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.101489 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3939775_883b_454d_9e3b_6949f6984dfb.slice/crio-6795c510085a438fb9965116ccfadbd399a6ade6b86b58ac66a031a8aa14aecf WatchSource:0}: Error finding container 6795c510085a438fb9965116ccfadbd399a6ade6b86b58ac66a031a8aa14aecf: Status 404 returned error can't find the container with id 6795c510085a438fb9965116ccfadbd399a6ade6b86b58ac66a031a8aa14aecf Nov 03 22:22:27.146223 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.146197 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hdlkm"] Nov 03 22:22:27.159416 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.159377 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ea235a_c0ba_44b9_909f_e3dd472a6c36.slice/crio-f86ef3f4fa9f6d1db1a27cd235dfd4bcc78a9105f6e3862064dfc0493b834c8f WatchSource:0}: Error finding container f86ef3f4fa9f6d1db1a27cd235dfd4bcc78a9105f6e3862064dfc0493b834c8f: Status 404 returned error can't find the container with id f86ef3f4fa9f6d1db1a27cd235dfd4bcc78a9105f6e3862064dfc0493b834c8f Nov 03 22:22:27.182643 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.182613 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6"] Nov 03 22:22:27.186337 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.186290 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad3ef2b_1bf1_41ec_8848_a2ffd108fb34.slice/crio-a1ad8cc30fd5c61fd78a184c67e09459ad82764d7840fe2b211856aaa07994e7 WatchSource:0}: Error finding container a1ad8cc30fd5c61fd78a184c67e09459ad82764d7840fe2b211856aaa07994e7: Status 404 returned error can't find the container with id a1ad8cc30fd5c61fd78a184c67e09459ad82764d7840fe2b211856aaa07994e7 Nov 03 22:22:27.190389 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.190355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:27.190498 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.190479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:27.190571 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.190546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:27.190755 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.190737 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:27.190836 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.190822 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.190781472 +0000 UTC m=+33.856891997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:27.190899 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.190892 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:27.190966 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.190929 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.190918279 +0000 UTC m=+33.857028446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:27.191027 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.190991 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:27.191027 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.191002 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:27.191163 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.191035 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.191022551 +0000 UTC m=+33.857132717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:27.205586 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.205519 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z5lhd"] Nov 03 22:22:27.212682 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.212657 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5210f23_6d88_413f_b0b5_c1b938713ca9.slice/crio-2cc6466a6ca067e1ac7162761d63058ae27fd4fd442195ff24253063ba9e40dc WatchSource:0}: Error finding container 2cc6466a6ca067e1ac7162761d63058ae27fd4fd442195ff24253063ba9e40dc: Status 404 returned error can't find the container with id 2cc6466a6ca067e1ac7162761d63058ae27fd4fd442195ff24253063ba9e40dc Nov 03 22:22:27.261874 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.261842 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs"] Nov 03 22:22:27.265074 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.265047 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fe9b11_1b54_4871_9236_dae9154b43fb.slice/crio-7058c42487b134a7e407662f1367cf2426cca7a8b7a3cbf07000fcb94d98e7dc WatchSource:0}: Error finding container 7058c42487b134a7e407662f1367cf2426cca7a8b7a3cbf07000fcb94d98e7dc: Status 404 returned error can't find the container with id 7058c42487b134a7e407662f1367cf2426cca7a8b7a3cbf07000fcb94d98e7dc Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.291135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.291170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.291224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.291265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291374 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291420 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.291407788 +0000 UTC m=+33.957517951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291463 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291482 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.291476525 +0000 UTC m=+33.957586688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291522 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.291516162 +0000 UTC m=+33.957626325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291561 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291568 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:27.291600 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.291585 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.29158043 +0000 UTC m=+33.957690593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:27.327371 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.327268 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-thjpw"] Nov 03 22:22:27.329935 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.329902 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a1808a_9488_4d49_8028_4860660c0c9d.slice/crio-acf959bc9242b1eaac3a5d71f53c19eaffa6993649deb71f5ddbae7427c0d433 WatchSource:0}: Error finding container acf959bc9242b1eaac3a5d71f53c19eaffa6993649deb71f5ddbae7427c0d433: Status 404 returned error can't find the container with id acf959bc9242b1eaac3a5d71f53c19eaffa6993649deb71f5ddbae7427c0d433 Nov 03 22:22:27.357572 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.357549 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5"] Nov 03 22:22:27.359681 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.359656 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f4d5dd_a512_425a_91b9_de7749f091c6.slice/crio-2b98dc03494e8dd3e1b4a96275c3281d260331648afb07927d568144c48df44b WatchSource:0}: Error finding container 2b98dc03494e8dd3e1b4a96275c3281d260331648afb07927d568144c48df44b: Status 404 returned error can't find the container with id 2b98dc03494e8dd3e1b4a96275c3281d260331648afb07927d568144c48df44b Nov 03 22:22:27.392369 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.392342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:27.392515 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.392496 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:27.392578 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:27.392564 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.392546811 +0000 UTC m=+34.058656967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:27.393653 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.393633 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqv9b"] Nov 03 22:22:27.396243 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:27.396216 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c95c406_8766_4a77_9639_513f2f37c1d3.slice/crio-2a5b40b6b115159914681045eafc12f661fa66018aa7af33537aa13109a3b71f WatchSource:0}: Error finding container 2a5b40b6b115159914681045eafc12f661fa66018aa7af33537aa13109a3b71f: Status 404 returned error can't find the container with id 2a5b40b6b115159914681045eafc12f661fa66018aa7af33537aa13109a3b71f Nov 03 22:22:27.530217 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.530186 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:27.786337 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.786260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:27.786337 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.786297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:27.786612 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.786350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:27.831842 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.831784 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:27.839480 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.839453 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:22:27.839670 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.839655 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2dn77\"" Nov 03 22:22:27.839731 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.839657 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kbfvs\"" Nov 03 22:22:27.839731 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.839674 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:22:27.839821 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.839728 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk9bw\"" Nov 03 22:22:27.994802 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.994760 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z5lhd" event={"ID":"d5210f23-6d88-413f-b0b5-c1b938713ca9","Type":"ContainerStarted","Data":"2cc6466a6ca067e1ac7162761d63058ae27fd4fd442195ff24253063ba9e40dc"} Nov 03 22:22:27.996224 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.996198 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" event={"ID":"35ea235a-c0ba-44b9-909f-e3dd472a6c36","Type":"ContainerStarted","Data":"b14fecc181667fae7199bce8dfbf1a72cc008b0b95d9b2de715080b944e558a8"} Nov 03 22:22:27.996354 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.996244 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" event={"ID":"35ea235a-c0ba-44b9-909f-e3dd472a6c36","Type":"ContainerStarted","Data":"f86ef3f4fa9f6d1db1a27cd235dfd4bcc78a9105f6e3862064dfc0493b834c8f"} Nov 03 22:22:27.997621 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.997596 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" event={"ID":"e3939775-883b-454d-9e3b-6949f6984dfb","Type":"ContainerStarted","Data":"ee5dba5bdf9afdfc722f8e097fc79593fce2765ed3d2fb12dc43b2d62da3e5ba"} Nov 03 22:22:27.997736 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.997635 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" event={"ID":"e3939775-883b-454d-9e3b-6949f6984dfb","Type":"ContainerStarted","Data":"6795c510085a438fb9965116ccfadbd399a6ade6b86b58ac66a031a8aa14aecf"} Nov 03 22:22:27.998783 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.998762 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" event={"ID":"b6fe9b11-1b54-4871-9236-dae9154b43fb","Type":"ContainerStarted","Data":"7058c42487b134a7e407662f1367cf2426cca7a8b7a3cbf07000fcb94d98e7dc"} Nov 03 22:22:27.999836 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:27.999808 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqv9b" event={"ID":"7c95c406-8766-4a77-9639-513f2f37c1d3","Type":"ContainerStarted","Data":"2a5b40b6b115159914681045eafc12f661fa66018aa7af33537aa13109a3b71f"} Nov 03 22:22:28.000977 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.000958 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" event={"ID":"65f4d5dd-a512-425a-91b9-de7749f091c6","Type":"ContainerStarted","Data":"2b98dc03494e8dd3e1b4a96275c3281d260331648afb07927d568144c48df44b"} Nov 03 22:22:28.001992 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.001970 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" event={"ID":"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34","Type":"ContainerStarted","Data":"a1ad8cc30fd5c61fd78a184c67e09459ad82764d7840fe2b211856aaa07994e7"} Nov 03 22:22:28.003080 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.003062 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" event={"ID":"e3a1808a-9488-4d49-8028-4860660c0c9d","Type":"ContainerStarted","Data":"acf959bc9242b1eaac3a5d71f53c19eaffa6993649deb71f5ddbae7427c0d433"} Nov 03 22:22:28.004402 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.004368 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" event={"ID":"e0a3e17d-abf7-4d4b-afe0-fd028d2caeff","Type":"ContainerStarted","Data":"3494b69bd4ee2440973f94c243ee4389e8f42a366683b0ee64bc098b08605e94"} Nov 03 22:22:28.004476 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.004421 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" event={"ID":"e0a3e17d-abf7-4d4b-afe0-fd028d2caeff","Type":"ContainerStarted","Data":"478ac8d747bd642b9bfb5ed1136dbc4c3b8ed747122f7dbeb5923b0d57f05ac0"} Nov 03 22:22:28.005395 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.005378 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" event={"ID":"3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34","Type":"ContainerStarted","Data":"23bfb68550386e57bbfef9fc5811d99c33613e35a85be46da473af252806b87c"} Nov 03 22:22:28.198771 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.198740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:28.198957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.198851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:28.198957 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.198894 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:28.198957 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.198915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:28.198957 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.198956 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.198939006 +0000 UTC m=+35.865049178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:28.199168 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.198989 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:28.199168 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.199009 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:28.199168 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.199033 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:28.199168 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.199042 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.19903093 +0000 UTC m=+35.865141093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:28.199168 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.199075 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.199062979 +0000 UTC m=+35.865173153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:28.299773 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.299736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:28.299964 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.299833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:28.299964 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.299876 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:28.299964 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.299896 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:28.299964 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.299954 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:28.299964 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.299963 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.299947626 +0000 UTC m=+35.966057776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:28.300146 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.299996 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.299986586 +0000 UTC m=+35.966096736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:28.300146 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.299881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:28.300146 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.299955 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:28.300146 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.300032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:28.300146 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.300060 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.300049585 +0000 UTC m=+35.966159767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:28.300146 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.300119 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.300111491 +0000 UTC m=+35.966221641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:28.378699 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.378663 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:28.400832 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.400798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:28.400990 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.400927 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:28.401049 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.400997 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.400977544 +0000 UTC m=+36.067087694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:28.499447 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.499342 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" podStartSLOduration=391.499322116 podStartE2EDuration="6m31.499322116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:28.497639867 +0000 UTC m=+34.163750040" watchObservedRunningTime="2026-11-03 22:22:28.499322116 +0000 UTC m=+34.165432291" Nov 03 22:22:28.603824 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.603643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:28.604027 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.603883 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:28.604027 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:28.603932 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:23:00.603918246 +0000 UTC m=+66.270028397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : secret "metrics-daemon-secret" not found Nov 03 22:22:28.620378 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.620345 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:28.704548 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.704516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:28.707753 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.707728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:28.719511 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.719478 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:28.733936 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.733886 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" podStartSLOduration=391.73386885 podStartE2EDuration="6m31.73386885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:28.733348027 +0000 UTC m=+34.399458201" watchObservedRunningTime="2026-11-03 22:22:28.73386885 +0000 UTC m=+34.399979022" Nov 03 22:22:28.735534 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.735490 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" podStartSLOduration=391.735480834 podStartE2EDuration="6m31.735480834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:28.617954643 +0000 UTC m=+34.284064815" watchObservedRunningTime="2026-11-03 22:22:28.735480834 +0000 UTC m=+34.401590983" Nov 03 22:22:28.805835 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.805714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:28.808995 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.808963 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:28.823560 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:28.823044 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:29.060113 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:29.059826 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:29.106754 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:29.106614 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:29.161318 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:29.161134 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c9btg"] Nov 03 22:22:29.490241 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:29.485446 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mzwbr"] Nov 03 22:22:29.918351 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:29.918310 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:30.022269 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.022118 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mzwbr" event={"ID":"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d","Type":"ContainerStarted","Data":"79e643b9bcc8e215e963ff27d4255c28ec8f908a97d1b541ba261eac021b9907"} Nov 03 22:22:30.024094 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.023864 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c9btg" event={"ID":"06b3993b-eaa8-4649-9c38-2b9bb7a22498","Type":"ContainerStarted","Data":"53db53896889d659be7345968316c874f3162d7f37c7186f83ee26b6b7a56b2c"} Nov 03 22:22:30.160497 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.160445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:30.221051 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.220991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:30.221127 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.221063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:30.221182 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.221161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:30.221409 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.221362 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:30.221478 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.221458 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.221438288 +0000 UTC m=+39.887548441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:30.221546 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.221525 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:30.221546 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.221535 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:30.221654 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.221569 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.221556567 +0000 UTC m=+39.887666724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:30.221654 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.221630 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:30.221776 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.221665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.221654704 +0000 UTC m=+39.887764860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:30.322222 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.322191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:30.322378 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.322273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:30.322378 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.322333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:30.322378 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.322374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:30.322582 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322567 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.32254841 +0000 UTC m=+39.988658563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:30.322684 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322672 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:30.322737 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322689 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:30.322737 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322724 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.322711444 +0000 UTC m=+39.988821614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:30.322861 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322803 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:30.322861 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322838 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.32282745 +0000 UTC m=+39.988937617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:30.323007 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322897 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:30.323007 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.322927 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.322916942 +0000 UTC m=+39.989027097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:30.358799 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.358726 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:30.423319 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:30.423235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:30.423472 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.423407 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:30.423537 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:30.423516 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:34.423456907 +0000 UTC m=+40.089567064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:31.031369 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:31.031333 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" event={"ID":"b6fe9b11-1b54-4871-9236-dae9154b43fb","Type":"ContainerStarted","Data":"dd8734daf6bbc29db2af0048d8de3a73fa244804702fd9c302bb2e8dc8c91705"} Nov 03 22:22:31.149687 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:31.149620 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" podStartSLOduration=361.200559784 podStartE2EDuration="6m4.149601358s" totalImagesPullingTime="2.949041574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:31.147986917 +0000 UTC m=+36.814097091" watchObservedRunningTime="2026-11-03 22:22:31.149601358 +0000 UTC m=+36.815711533" Nov 03 22:22:31.295164 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:31.295064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.494200 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:31.494165 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.693685 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:31.693605 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.890417 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:31.890388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:33.046457 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:33.046407 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/0.log" Nov 03 22:22:33.046457 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:33.046446 2578 generic.go:356] "Generic (PLEG): container finished" podID="e3a1808a-9488-4d49-8028-4860660c0c9d" containerID="f907d9733e1b35da6743e3e457fdca9f4d79196aad2304955d24e6a06d5a3685" exitCode=255 Nov 03 22:22:33.046970 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:33.046732 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" event={"ID":"e3a1808a-9488-4d49-8028-4860660c0c9d","Type":"ContainerDied","Data":"f907d9733e1b35da6743e3e457fdca9f4d79196aad2304955d24e6a06d5a3685"} Nov 03 22:22:33.046970 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:33.046883 2578 scope.go:122] "RemoveContainer" containerID="f907d9733e1b35da6743e3e457fdca9f4d79196aad2304955d24e6a06d5a3685" Nov 03 22:22:33.354131 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:33.354078 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:33.611208 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:33.611177 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:34.063412 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.063371 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqv9b" event={"ID":"7c95c406-8766-4a77-9639-513f2f37c1d3","Type":"ContainerStarted","Data":"f14aba89849c06356572ab4c92e7fc5d8a39b69c69accdcba2f3f5bdec83c255"} Nov 03 22:22:34.066256 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.066229 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/1.log" Nov 03 22:22:34.066749 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.066724 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/0.log" Nov 03 22:22:34.066864 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.066766 2578 generic.go:356] "Generic (PLEG): container finished" podID="e3a1808a-9488-4d49-8028-4860660c0c9d" containerID="fbf7747233d82ae2ff25e3a729e1ddf99197402b0e6cff13442a2475f1cc27bf" exitCode=255 Nov 03 22:22:34.067105 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.067080 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" event={"ID":"e3a1808a-9488-4d49-8028-4860660c0c9d","Type":"ContainerDied","Data":"fbf7747233d82ae2ff25e3a729e1ddf99197402b0e6cff13442a2475f1cc27bf"} Nov 03 22:22:34.067192 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.067156 2578 scope.go:122] "RemoveContainer" containerID="f907d9733e1b35da6743e3e457fdca9f4d79196aad2304955d24e6a06d5a3685" Nov 03 22:22:34.067245 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.067195 2578 scope.go:122] "RemoveContainer" containerID="fbf7747233d82ae2ff25e3a729e1ddf99197402b0e6cff13442a2475f1cc27bf" Nov 03 22:22:34.068068 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.068040 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-thjpw_openshift-console-operator(e3a1808a-9488-4d49-8028-4860660c0c9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" podUID="e3a1808a-9488-4d49-8028-4860660c0c9d" Nov 03 22:22:34.069551 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.069526 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z5lhd" event={"ID":"d5210f23-6d88-413f-b0b5-c1b938713ca9","Type":"ContainerStarted","Data":"a844ae340dbed8549b157f1dbacbfbc254d0e95bf71f727d0da9a6792ff74020"} Nov 03 22:22:34.262276 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.262244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:34.262439 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.262320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:34.262439 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.262391 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:34.262439 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.262407 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:34.262439 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.262414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:34.262673 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.262461 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.262444117 +0000 UTC m=+47.928554287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:34.262673 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.262508 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:34.262673 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.262545 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.262534965 +0000 UTC m=+47.928645128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:34.262673 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.262595 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:34.262673 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.262626 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.262615997 +0000 UTC m=+47.928726161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:34.302304 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.302256 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pqv9b" podStartSLOduration=2.150217933 podStartE2EDuration="8.302243254s" totalImagesPullingTime="6.152025321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:34.182415727 +0000 UTC m=+39.848525922" watchObservedRunningTime="2026-11-03 22:22:34.302243254 +0000 UTC m=+39.968353427" Nov 03 22:22:34.363301 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.363262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:34.363474 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.363370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:34.363474 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363400 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.363381675 +0000 UTC m=+48.029491838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:34.363474 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363461 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:34.363474 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363475 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:34.363676 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.363475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:34.363676 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363519 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.363504823 +0000 UTC m=+48.029614989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:34.363676 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.363564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:34.363676 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363572 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:34.363676 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363608 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.363599412 +0000 UTC m=+48.029709574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:34.363902 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363693 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:34.363902 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.363756 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.363739803 +0000 UTC m=+48.029849959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:34.419159 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.419108 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-z5lhd" podStartSLOduration=361.723363201 podStartE2EDuration="6m7.419094859s" totalImagesPullingTime="5.695731658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:34.41871194 +0000 UTC m=+40.084822111" watchObservedRunningTime="2026-11-03 22:22:34.419094859 +0000 UTC m=+40.085205033" Nov 03 22:22:34.464669 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.464634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:34.464831 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.464778 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:34.464877 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.464865 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.464846453 +0000 UTC m=+48.130956609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:34.746730 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.746683 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:34.755040 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.755012 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r8md2"] Nov 03 22:22:34.776960 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.776934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.815077 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.815048 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r8md2"] Nov 03 22:22:34.829984 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.829693 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:22:34.829984 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.829817 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:22:34.830382 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.830363 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vkjzf\"" Nov 03 22:22:34.868446 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.868376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cc81f17-d52c-4edb-9f33-de2f37c80b54-data-volume\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.868591 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.868445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwpg\" (UniqueName: \"kubernetes.io/projected/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-api-access-qqwpg\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.868591 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.868512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.868716 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.868629 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7cc81f17-d52c-4edb-9f33-de2f37c80b54-crio-socket\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.868716 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.868703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.969153 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.969113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7cc81f17-d52c-4edb-9f33-de2f37c80b54-crio-socket\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.969306 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.969175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.969306 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.969266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cc81f17-d52c-4edb-9f33-de2f37c80b54-data-volume\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.969416 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.969319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwpg\" (UniqueName: \"kubernetes.io/projected/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-api-access-qqwpg\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.969416 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.969335 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:34.969416 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.969406 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:35.469386478 +0000 UTC m=+41.135496647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : configmap "kube-rbac-proxy" not found Nov 03 22:22:34.969416 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.969339 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7cc81f17-d52c-4edb-9f33-de2f37c80b54-crio-socket\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.969668 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.969345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.969668 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.969413 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:34.969668 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:34.969509 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:35.469497068 +0000 UTC m=+41.135607231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:34.969668 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.969657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cc81f17-d52c-4edb-9f33-de2f37c80b54-data-volume\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:34.988328 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:34.988289 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:35.022303 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.022269 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fnl7t-pdbr9"] Nov 03 22:22:35.032930 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.032892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwpg\" (UniqueName: \"kubernetes.io/projected/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-api-access-qqwpg\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:35.040319 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.040295 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.070972 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.070884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.070972 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.070931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.071399 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.071021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x762b\" (UniqueName: \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.071399 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.071287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.076601 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.075478 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" event={"ID":"65f4d5dd-a512-425a-91b9-de7749f091c6","Type":"ContainerStarted","Data":"6e7a9a5fff1a9b5e3b191f58c3008c5b45b77ccb096a44b3aec96b0f450df434"} Nov 03 22:22:35.080723 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.080112 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/1.log" Nov 03 22:22:35.080723 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.080480 2578 scope.go:122] "RemoveContainer" containerID="fbf7747233d82ae2ff25e3a729e1ddf99197402b0e6cff13442a2475f1cc27bf" Nov 03 22:22:35.080723 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:35.080655 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-thjpw_openshift-console-operator(e3a1808a-9488-4d49-8028-4860660c0c9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" podUID="e3a1808a-9488-4d49-8028-4860660c0c9d" Nov 03 22:22:35.081963 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.081939 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fnl7t-pdbr9"] Nov 03 22:22:35.083779 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.083754 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" event={"ID":"3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34","Type":"ContainerStarted","Data":"a08d6b4c14c9a13a2a9745b6a3af4ccaaacc69f5d158307551dd8628ee3b5890"} Nov 03 22:22:35.089303 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.088942 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mzwbr" event={"ID":"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d","Type":"ContainerStarted","Data":"38ce6ca42f0b634824ed537091ccddb51625c2c1389da26a158eacda636371a9"} Nov 03 22:22:35.089303 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.089137 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:35.173013 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.172981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.173150 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.173051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.173150 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.173098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.173150 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.173139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x762b\" (UniqueName: \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.174733 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.174704 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.175263 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.175240 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.176357 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.176310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.189269 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.189239 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:35.236802 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.236762 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x762b\" (UniqueName: \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.275368 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.275310 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" podStartSLOduration=360.840650871 podStartE2EDuration="6m8.275291124s" totalImagesPullingTime="7.434640253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:35.273469223 +0000 UTC m=+40.939579396" watchObservedRunningTime="2026-11-03 22:22:35.275291124 +0000 UTC m=+40.941401301" Nov 03 22:22:35.365303 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.365261 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:35.476570 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.476539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:35.476693 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.476621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:35.476693 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:35.476655 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:35.476758 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:35.476715 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:36.476700706 +0000 UTC m=+42.142810870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:35.476758 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:35.476718 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:35.476883 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:35.476764 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:36.476754165 +0000 UTC m=+42.142864315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : configmap "kube-rbac-proxy" not found Nov 03 22:22:35.512273 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.512215 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" podStartSLOduration=361.324757928 podStartE2EDuration="6m8.512205251s" totalImagesPullingTime="7.187447323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:35.511946263 +0000 UTC m=+41.178056437" watchObservedRunningTime="2026-11-03 22:22:35.512205251 +0000 UTC m=+41.178315422" Nov 03 22:22:35.630379 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.630317 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mzwbr" podStartSLOduration=34.387956686 podStartE2EDuration="39.630297779s" totalImagesPullingTime="5.242341093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:35.628854919 +0000 UTC m=+41.294965093" watchObservedRunningTime="2026-11-03 22:22:35.630297779 +0000 UTC m=+41.296407954" Nov 03 22:22:35.774615 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.774589 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fnl7t-pdbr9"] Nov 03 22:22:35.777323 ip-10-0-139-173 kubenswrapper[2578]: W1103 22:22:35.777290 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0322b8e_5e35_4e52_b5dd_42f32573354f.slice/crio-2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe WatchSource:0}: Error finding container 2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe: Status 404 returned error can't find the container with id 2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe Nov 03 22:22:35.981608 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:35.981573 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.093626 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.093587 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" event={"ID":"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34","Type":"ContainerStarted","Data":"d762dfe0c536292aa8e8a2de58744cdd600913e58b5205a7ded54c9829074d14"} Nov 03 22:22:36.095371 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.095340 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" event={"ID":"e0322b8e-5e35-4e52-b5dd-42f32573354f","Type":"ContainerStarted","Data":"990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1"} Nov 03 22:22:36.095499 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.095391 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" event={"ID":"e0322b8e-5e35-4e52-b5dd-42f32573354f","Type":"ContainerStarted","Data":"2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe"} Nov 03 22:22:36.177768 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.177734 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.210893 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.210822 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" podStartSLOduration=361.350444338 podStartE2EDuration="6m9.210801571s" totalImagesPullingTime="7.860357233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:36.210426538 +0000 UTC m=+41.876536713" watchObservedRunningTime="2026-11-03 22:22:36.210801571 +0000 UTC m=+41.876911741" Nov 03 22:22:36.333392 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.333339 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" podStartSLOduration=2.333319616 podStartE2EDuration="2.333319616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:36.332128313 +0000 UTC m=+41.998238486" watchObservedRunningTime="2026-11-03 22:22:36.333319616 +0000 UTC m=+41.999429788" Nov 03 22:22:36.484897 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.484865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:36.485007 ip-10-0-139-173 kubenswrapper[2578]: I1103 22:22:36.484982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:36.485072 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:36.485019 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:36.485110 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:36.485082 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.485061851 +0000 UTC m=+44.151172018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:36.485110 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:36.485092 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:36.485211 ip-10-0-139-173 kubenswrapper[2578]: E1103 22:22:36.485130 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.485117838 +0000 UTC m=+44.151228008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : configmap "kube-rbac-proxy" not found Nov 03 22:22:36.720200 ip-10-0-139-173 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:22:36.742238 ip-10-0-139-173 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:22:36.742406 ip-10-0-139-173 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:22:36.742472 ip-10-0-139-173 systemd[1]: kubelet.service: Consumed 3.083s CPU time. Nov 03 22:22:36.754509 ip-10-0-139-173 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:22:36.817157 ip-10-0-139-173 kubenswrapper[4898]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:36.817157 ip-10-0-139-173 kubenswrapper[4898]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:22:36.817157 ip-10-0-139-173 kubenswrapper[4898]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:36.817157 ip-10-0-139-173 kubenswrapper[4898]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:36.819363 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819284 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.819363 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819362 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.819363 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819366 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819370 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819373 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819376 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819379 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819382 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819385 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819387 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819390 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819393 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819396 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819398 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819407 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819410 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819413 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819416 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819418 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819421 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819424 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.819473 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819427 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819429 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819432 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819435 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819437 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819440 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819444 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819449 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819453 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819456 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819459 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819462 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819465 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819468 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819470 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819473 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819476 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819479 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819481 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.819985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819484 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819487 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819489 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819493 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819496 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819498 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819501 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819504 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819507 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819510 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819513 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819516 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819518 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819521 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819524 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819526 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819529 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819532 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819535 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819537 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.820441 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819541 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819544 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819547 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819549 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819552 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819555 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819557 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819560 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819563 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819565 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819568 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819571 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819573 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819576 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819579 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819581 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819584 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819587 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819590 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819592 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.820996 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819596 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819599 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819602 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819604 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819607 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819609 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819612 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819618 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819620 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819623 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819627 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819630 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819634 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819637 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819639 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819642 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819645 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819648 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819651 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.821491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819654 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819657 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819660 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819663 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819665 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819668 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.819671 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820423 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820430 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820433 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820437 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820440 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820443 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820446 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820449 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820452 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820455 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820457 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820460 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.821986 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820463 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820466 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820468 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820471 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820474 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820476 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820479 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820482 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820485 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820488 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820491 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820493 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820496 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820498 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820501 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820504 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820507 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820509 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820513 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.822436 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820517 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820520 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820523 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820526 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820528 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820532 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820534 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820537 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820541 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820546 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820549 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820552 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820554 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820557 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820559 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820562 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820565 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820568 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820571 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.822921 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820573 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820576 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820578 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820581 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820584 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820588 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820590 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820593 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820596 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820599 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820601 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820604 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820607 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820610 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820613 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820616 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820618 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820621 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820624 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.823388 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820627 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820630 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820633 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820636 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820639 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820641 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820644 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820646 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820649 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820651 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820656 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820659 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820662 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820664 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820667 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820670 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820673 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820675 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820678 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820681 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.823866 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820683 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820686 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820688 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820691 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820694 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820696 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820699 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820702 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820704 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820707 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820710 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820712 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820714 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820717 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820720 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820723 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.820727 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821221 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821227 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821230 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.824352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821233 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821236 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821241 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821243 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821246 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821249 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821251 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821255 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821257 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821260 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821262 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821266 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821268 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821272 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821275 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821277 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821280 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821283 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821286 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.824881 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821290 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821294 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821297 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821301 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821303 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821306 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821309 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821312 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821315 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821317 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821320 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821323 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821325 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821328 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821331 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821334 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821337 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821341 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821345 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.825369 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821349 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821352 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821355 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821358 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821361 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821364 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821367 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821372 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821374 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821377 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821380 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821383 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821385 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821388 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821390 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821393 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821395 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821398 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821401 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821404 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.825860 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821407 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821409 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821412 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821414 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821417 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821420 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821422 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821425 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821428 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821431 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821434 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821437 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821440 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821443 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821445 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821448 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821451 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821453 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821456 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821460 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.826352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821463 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821465 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821468 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821470 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821473 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821476 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821478 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821481 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821483 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821486 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821488 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821492 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821494 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821497 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821500 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821503 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821506 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821508 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821511 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.826883 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821513 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821516 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821519 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821522 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821525 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.821528 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821618 4898 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821625 4898 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821632 4898 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821637 4898 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821642 4898 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821645 4898 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821651 4898 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821655 4898 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821658 4898 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821662 4898 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821665 4898 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821669 4898 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821672 4898 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821675 4898 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821678 4898 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821681 4898 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821684 4898 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:22:36.827348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821687 4898 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821691 4898 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821694 4898 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821697 4898 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821701 4898 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821705 4898 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821709 4898 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821712 4898 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821715 4898 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821718 4898 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821722 4898 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821725 4898 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821728 4898 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821732 4898 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821735 4898 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821739 4898 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821742 4898 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821746 4898 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821749 4898 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821752 4898 flags.go:64] FLAG: --enable-server="true" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821754 4898 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821760 4898 flags.go:64] FLAG: --event-burst="100" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821764 4898 flags.go:64] FLAG: --event-qps="50" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821767 4898 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:22:36.827920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821770 4898 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821773 4898 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821777 4898 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821781 4898 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821796 4898 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821799 4898 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821803 4898 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821806 4898 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821809 4898 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821813 4898 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821816 4898 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821819 4898 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821822 4898 flags.go:64] FLAG: --feature-gates="" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821829 4898 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821833 4898 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821836 4898 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821839 4898 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821843 4898 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821846 4898 flags.go:64] FLAG: --help="false" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821850 4898 flags.go:64] FLAG: --hostname-override="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821853 4898 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821856 4898 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821859 4898 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:22:36.828526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821863 4898 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821866 4898 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821870 4898 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821873 4898 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821876 4898 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821879 4898 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821883 4898 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821887 4898 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821889 4898 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821892 4898 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821896 4898 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821899 4898 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821902 4898 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821905 4898 flags.go:64] FLAG: --lock-file="" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821908 4898 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821911 4898 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821914 4898 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821920 4898 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821923 4898 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821926 4898 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821929 4898 flags.go:64] FLAG: --logging-format="text" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821932 4898 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821935 4898 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821939 4898 flags.go:64] FLAG: --manifest-url="" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821942 4898 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:22:36.829373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821947 4898 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821950 4898 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821954 4898 flags.go:64] FLAG: --max-pods="110" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821957 4898 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821960 4898 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821963 4898 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821966 4898 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821970 4898 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821973 4898 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821976 4898 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821983 4898 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821986 4898 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821990 4898 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821994 4898 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.821997 4898 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822000 4898 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822003 4898 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822008 4898 flags.go:64] FLAG: --port="10250" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822011 4898 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822014 4898 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d594187a7cebf8f0" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822017 4898 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822020 4898 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822024 4898 flags.go:64] FLAG: --register-node="true" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822027 4898 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822030 4898 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822033 4898 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:22:36.830447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822036 4898 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822039 4898 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822044 4898 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822047 4898 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822050 4898 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822054 4898 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822057 4898 flags.go:64] FLAG: --runonce="false" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822060 4898 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822063 4898 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822066 4898 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822069 4898 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822072 4898 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822076 4898 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822079 4898 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822082 4898 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822085 4898 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822088 4898 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822091 4898 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822094 4898 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822098 4898 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822101 4898 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822104 4898 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822109 4898 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822114 4898 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822117 4898 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:22:36.831175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822120 4898 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822123 4898 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822126 4898 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822129 4898 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822132 4898 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822135 4898 flags.go:64] FLAG: --v="2" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822139 4898 flags.go:64] FLAG: --version="false" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822143 4898 flags.go:64] FLAG: --vmodule="" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822147 4898 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822151 4898 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822276 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822280 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822283 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822286 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822304 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822308 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822311 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822314 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822317 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822320 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822324 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822328 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.831859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822331 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822335 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822337 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822340 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822345 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822347 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822351 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822354 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822358 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822360 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822363 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822366 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822369 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822371 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822374 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822377 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822379 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822382 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822384 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822387 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.832408 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822389 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822392 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822395 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822398 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822401 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822404 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822406 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822409 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822413 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822418 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822421 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822424 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822427 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822430 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822432 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822435 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822439 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822442 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822445 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.833073 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822448 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822451 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822454 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822457 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822459 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822462 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822465 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822467 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822470 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822473 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822475 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822478 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822481 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822483 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822486 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822489 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822492 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822494 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822497 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.833850 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822500 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822502 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822505 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822507 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822510 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822512 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822515 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822518 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822520 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822523 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822527 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822530 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822533 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822535 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822539 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822542 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822545 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822548 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822550 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822553 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.834352 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822555 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822558 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822561 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822563 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822566 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822568 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822571 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822574 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822576 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822579 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822581 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822584 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822587 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822590 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822592 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822595 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.834982 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.822620 4898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822655 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822660 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822662 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822665 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822668 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822671 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822674 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822677 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822680 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822684 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822687 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822690 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822694 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822698 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822701 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822704 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822706 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822709 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822712 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.835413 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822715 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822717 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822720 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822723 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822725 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822728 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822731 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822733 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822736 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822739 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822742 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822745 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822747 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822750 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822752 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822755 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822758 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822761 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822764 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822769 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.836149 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822771 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822774 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822778 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822782 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822798 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822802 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822805 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822808 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822811 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822814 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822816 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822819 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822821 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822824 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822826 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822829 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822832 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822834 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822838 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.836733 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822840 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822843 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822845 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822849 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822851 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822854 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822857 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822859 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822862 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822865 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822868 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822871 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822877 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822879 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822882 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822886 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822889 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822891 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822894 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822897 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.837242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822899 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822902 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822905 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822907 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822910 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822913 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822915 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822918 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822920 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822923 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822926 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822929 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822932 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822935 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822937 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822940 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822943 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822946 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822949 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.837719 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822951 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822954 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822956 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822959 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822961 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822965 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822967 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822970 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.822975 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.827095 4898 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.827108 4898 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.827127 4898 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.827131 4898 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827216 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827221 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827224 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827228 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827231 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827234 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.838179 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827236 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827239 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827241 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827244 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827247 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827250 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827253 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827255 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827258 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827261 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827270 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827273 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827275 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827278 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827281 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827284 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827287 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827290 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827292 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827295 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.838629 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827298 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827301 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827304 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827306 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827309 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827312 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827315 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827317 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827320 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827322 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827325 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827328 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827330 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827333 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827335 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827338 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827340 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827343 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827346 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827348 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.839208 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827351 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827355 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827358 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827366 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827369 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827372 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827375 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827377 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827380 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827382 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827385 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827388 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827391 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827393 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827396 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827398 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827402 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827407 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.839696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827411 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827415 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827418 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827420 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827423 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827426 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827428 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827431 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827434 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827437 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827439 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827442 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827445 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827447 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827450 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827452 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827455 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827458 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827466 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827470 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.840145 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827472 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827475 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827478 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827480 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827483 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827486 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827488 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827491 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827493 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827496 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827499 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827501 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827504 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827506 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827508 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827511 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827514 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827516 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827519 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827521 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.840665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827524 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827527 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.827553 4898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827741 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827746 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827749 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827752 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827754 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827757 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827759 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827762 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827770 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827773 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827776 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827780 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827797 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.841164 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827803 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827813 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827817 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827820 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827823 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827826 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827829 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827831 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827834 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827837 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827840 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827842 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827845 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827848 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827850 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827854 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827858 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827861 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827864 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.841545 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827867 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827870 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827873 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827876 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827878 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827881 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827883 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827886 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827889 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827897 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827900 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827903 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827906 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827908 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827911 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827914 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827916 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827919 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827922 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.842014 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827924 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827927 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827930 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827933 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827936 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827939 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827941 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827944 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827947 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827949 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827952 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827954 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827957 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827959 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827962 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827965 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827967 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827970 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827972 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827975 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.842487 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827977 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827980 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827982 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827991 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827995 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.827997 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828000 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828002 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828005 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828007 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828010 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828013 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828019 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828022 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828025 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828027 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828030 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828033 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828035 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.843261 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828038 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828041 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828043 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828046 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828049 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828051 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828054 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828056 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828059 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828062 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828064 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828067 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828070 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828072 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828075 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828078 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.843878 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.828103 4898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828162 4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828167 4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828170 4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828173 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828176 4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828179 4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828181 4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828184 4898 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828187 4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828191 4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828194 4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828196 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828199 4898 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828202 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828204 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828207 4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828210 4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828212 4898 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828215 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:36.844364 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828217 4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828220 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828223 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828225 4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828228 4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828230 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828233 4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828236 4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828238 4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828241 4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828244 4898 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828246 4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828249 4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828252 4898 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828260 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828263 4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828266 4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828268 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828271 4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:36.845007 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828273 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828276 4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828279 4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828283 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828286 4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828288 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828291 4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828293 4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828296 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828299 4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828301 4898 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828304 4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828307 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828309 4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828312 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828314 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828317 4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828320 4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828322 4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828325 4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:36.845648 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828327 4898 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828330 4898 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828333 4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828335 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828338 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828342 4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828346 4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828349 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828358 4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828361 4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828364 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828366 4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828369 4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828372 4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828374 4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828379 4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828382 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828385 4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828388 4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:36.846402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828390 4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828393 4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828396 4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828399 4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828403 4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828406 4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828409 4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828412 4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828415 4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828418 4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828421 4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828424 4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828426 4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828429 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828432 4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828435 4898 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828437 4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828440 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828442 4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:36.847254 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828446 4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828449 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828452 4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828460 4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828463 4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828466 4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828468 4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828471 4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828474 4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:36.828478 4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.828528 4898 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.830635 4898 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.830686 4898 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.830901 4898 server.go:1014] "Starting client certificate rotation" Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.831045 4898 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.831114 4898 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:16:55 +0000 UTC" deadline="2028-04-03 07:19:10.908611098 +0000 UTC" Nov 03 22:22:36.847938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.831156 4898 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12392h56m34.077459093s" Nov 03 22:22:36.848401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.831336 4898 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:36.848401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.832408 4898 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:36.848401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.834099 4898 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:22:36.848401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.835876 4898 log.go:25] "Validated CRI v1 image API" Nov 03 22:22:36.848401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.836680 4898 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:22:36.848401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.838671 4898 fs.go:135] Filesystem UUIDs: map[65fd09cf-bf74-4d6a-b9b2-90e88c405ca0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eaca8f28-a109-42f8-b0eb-4f0ab6413648:/dev/nvme0n1p4] Nov 03 22:22:36.848571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.838686 4898 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ee286eb53183ff3d40245d8fbb2be7ee833ac2bf8697cf5d45d12d63789033b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ee286eb53183ff3d40245d8fbb2be7ee833ac2bf8697cf5d45d12d63789033b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23bfb68550386e57bbfef9fc5811d99c33613e35a85be46da473af252806b87c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23bfb68550386e57bbfef9fc5811d99c33613e35a85be46da473af252806b87c/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a5b40b6b115159914681045eafc12f661fa66018aa7af33537aa13109a3b71f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a5b40b6b115159914681045eafc12f661fa66018aa7af33537aa13109a3b71f/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b98dc03494e8dd3e1b4a96275c3281d260331648afb07927d568144c48df44b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b98dc03494e8dd3e1b4a96275c3281d260331648afb07927d568144c48df44b/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cc6466a6ca067e1ac7162761d63058ae27fd4fd442195ff24253063ba9e40dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cc6466a6ca067e1ac7162761d63058ae27fd4fd442195ff24253063ba9e40dc/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/478ac8d747bd642b9bfb5ed1136dbc4c3b8ed747122f7dbeb5923b0d57f05ac0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/478ac8d747bd642b9bfb5ed1136dbc4c3b8ed747122f7dbeb5923b0d57f05ac0/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53db53896889d659be7345968316c874f3162d7f37c7186f83ee26b6b7a56b2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53db53896889d659be7345968316c874f3162d7f37c7186f83ee26b6b7a56b2c/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60b13fc0432176f1ed91db525e823ea8e00d1c18f237de750d591eebc3058a62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60b13fc0432176f1ed91db525e823ea8e00d1c18f237de750d591eebc3058a62/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60c55366c10392824707b262927dd6545f2141c7ca0ee2ae3b98efef3bc1d6fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60c55366c10392824707b262927dd6545f2141c7ca0ee2ae3b98efef3bc1d6fe/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6795c510085a438fb9965116ccfadbd399a6ade6b86b58ac66a031a8aa14aecf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6795c510085a438fb9965116ccfadbd399a6ade6b86b58ac66a031a8aa14aecf/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f366bb5ef5917b9fbc4d75946c8250753bac840431368aef2ef57a9768682a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f366bb5ef5917b9fbc4d75946c8250753bac840431368aef2ef57a9768682a5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7058c42487b134a7e407662f1367cf2426cca7a8b7a3cbf07000fcb94d98e7dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7058c42487b134a7e407662f1367cf2426cca7a8b7a3cbf07000fcb94d98e7dc/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79e643b9bcc8e215e963ff27d4255c28ec8f908a97d1b541ba261eac021b9907/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79e643b9bcc8e215e963ff27d4255c28ec8f908a97d1b541ba261eac021b9907/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1ad8cc30fd5c61fd78a184c67e09459ad82764d7840fe2b211856aaa07994e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1ad8cc30fd5c61fd78a184c67e09459ad82764d7840fe2b211856aaa07994e7/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acf959bc9242b1eaac3a5d71f53c19eaffa6993649deb71f5ddbae7427c0d433/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acf959bc9242b1eaac3a5d71f53c19eaffa6993649deb71f5ddbae7427c0d433/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b81844cd89060dd454ac71b792cc01cfdf446a2e5ff01e9dc4fb444d37946637/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b81844cd89060dd454ac71b792cc01cfdf446a2e5ff01e9dc4fb444d37946637/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e43502be7dc286327ba3930be1ff7900403ba713d6f0d2a5b50b2f2592c69a36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e43502be7dc286327ba3930be1ff7900403ba713d6f0d2a5b50b2f2592c69a36/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8bfc9f4803efea7ff26427281671784a21595fd246c625a963545f2036c2508/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8bfc9f4803efea7ff26427281671784a21595fd246c625a963545f2036c2508/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1ce9e0ec2670160ecf38165c74868eed7e49647387b153a664244b8a8fd82dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1ce9e0ec2670160ecf38165c74868eed7e49647387b153a664244b8a8fd82dd/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f86ef3f4fa9f6d1db1a27cd235dfd4bcc78a9105f6e3862064dfc0493b834c8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f86ef3f4fa9f6d1db1a27cd235dfd4bcc78a9105f6e3862064dfc0493b834c8f/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06b3993b-eaa8-4649-9c38-2b9bb7a22498/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/06b3993b-eaa8-4649-9c38-2b9bb7a22498/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~projected/kube-api-access-ghp7z:{mountpoint:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~projected/kube-api-access-ghp7z major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d7234dd-7b73-427d-b0e1-afd778bd2028/volumes/kubernetes.io~projected/kube-api-access-fxd8q:{mountpoint:/var/lib/kubelet/pods/1d7234dd-7b73-427d-b0e1-afd778bd2028/volumes/kubernetes.io~projected/kube-api-access-fxd8q major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f23b7eb-3f0e-4215-a42c-c20c976900dc/volumes/kubernetes.io~projected/kube-api-access-cgpc9:{mountpoint:/var/lib/kubelet/pods/1f23b7eb-3f0e-4215-a42c-c20c976900dc/volumes/kubernetes.io~projected/kube-api-access-cgpc9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35ea235a-c0ba-44b9-909f-e3dd472a6c36/volumes/kubernetes.io~projected/kube-api-access-cgqnx:{mountpoint:/var/lib/kubelet/pods/35ea235a-c0ba-44b9-909f-e3dd472a6c36/volumes/kubernetes.io~projected/kube-api-access-cgqnx major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~projected/kube-api-access-bzvnh:{mountpoint:/var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~projected/kube-api-access-bzvnh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~projected/kube-api-access-knc9r:{mountpoint:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~projected/kube-api-access-knc9r major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34/volumes/kubernetes.io~projected/kube-api-access-j9qrs:{mountpoint:/var/lib/kubelet/pods/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34/volumes/kubernetes.io~projected/kube-api-access-j9qrs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/458febd7-7dcd-4ef1-9b8b-27037557b753/volumes/kubernetes.io~projected/kube-api-access-9wjrh:{mountpoint:/var/lib/kubelet/pods/458febd7-7dcd-4ef1-9b8b-27037557b753/volumes/kubernetes.io~projected/kube-api-access-9wjrh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34/volumes/kubernetes.io~projected/kube-api-access-kpnfb:{mountpoint:/var/lib/kubelet/pods/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34/volumes/kubernetes.io~projected/kube-api-access-kpnfb major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60be4f8c-7051-444d-ad87-7e7895c94542/volumes/kubernetes.io~projected/kube-api-access-hpk4w:{mountpoint:/var/lib/kubelet/pods/60be4f8c-7051-444d-ad87-7e7895c94542/volumes/kubernetes.io~projected/kube-api-access-hpk4w major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64ebd614-43bd-479e-8e95-c33b2c93361c/volumes/kubernetes.io~projected/kube-api-access-g6xjb:{mountpoint:/var/lib/kubelet/pods/64ebd614-43bd-479e-8e95-c33b2c93361c/volumes/kubernetes.io~projected/kube-api-access-g6xjb major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65f4d5dd-a512-425a-91b9-de7749f091c6/volumes/kubernetes.io~projected/kube-api-access-4l29j:{mountpoint:/var/lib/kubelet/pods/65f4d5dd-a512-425a-91b9-de7749f091c6/volumes/kubernetes.io~projected/kube-api-access-4l29j major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65f4d5dd-a512-425a-91b9-de7749f091c6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/65f4d5dd-a512-425a-91b9-de7749f091c6/volumes/kubernetes.io~secret/serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68e1c673-629b-4840-9f7a-aa64777cf32a/volumes/kubernetes.io~projected/kube-api-access-j5jpx:{mountpoint:/var/lib/kubelet/pods/68e1c673-629b-4840-9f7a-aa64777cf32a/volumes/kubernetes.io~projected/kube-api-access-j5jpx major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~projected/kube-api-access-g5pmx:{mountpoint:/var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~projected/kube-api-access-g5pmx major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~secret/default-certificate major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~secret/stats-auth major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volumes/kubernetes.io~projected/kube-api-access-dsqkb:{mountpoint:/var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volumes/kubernetes.io~projected/kube-api-access-dsqkb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c95c406-8766-4a77-9639-513f2f37c1d3/volumes/kubernetes.io~projected/kube-api-access-sbfgm:{mountpoint:/var/lib/kubelet/pods/7c95c406-8766-4a77-9639-513f2f37c1d3/volumes/kubernetes.io~projected/kube-api-access-sbfgm major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c95c406-8766-4a77-9639-513f2f37c1d3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7c95c406-8766-4a77-9639-513f2f37c1d3/volumes/kubernetes.io~secret/cert major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cc81f17-d52c-4edb-9f33-de2f37c80b54/volumes/kubernetes.io~projected/kube-api-access-qqwpg:{mountpoint:/var/lib/kubelet/pods/7cc81f17-d52c-4edb-9f33-de2f37c80b54/volumes/kubernetes.io~projected/kube-api-access-qqwpg major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9b5b425-585c-4fc9-bc11-2510c7035c70/volumes/kubernetes.io~projected/kube-api-access-sx8xp:{mountpoint:/var/lib/kubelet/pods/a9b5b425-585c-4fc9-bc11-2510c7035c70/volumes/kubernetes.io~projected/kube-api-access-sx8xp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/add89f53-e47e-41de-8d0d-6fc06d826d29/volumes/kubernetes.io~projected/kube-api-access-dstxj:{mountpoint:/var/lib/kubelet/pods/add89f53-e47e-41de-8d0d-6fc06d826d29/volumes/kubernetes.io~projected/kube-api-access-dstxj major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6fe9b11-1b54-4871-9236-dae9154b43fb/volumes/kubernetes.io~projected/kube-api-access-4fh74:{mountpoint:/var/lib/kubelet/pods/b6fe9b11-1b54-4871-9236-dae9154b43fb/volumes/kubernetes.io~projected/kube-api-access-4fh74 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a374ff-7146-47f5-bcbd-6c4dc710d580/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c1a374ff-7146-47f5-bcbd-6c4dc710d580/volumes/kubernetes.io~secret/agent-certs major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5210f23-6d88-413f-b0b5-c1b938713ca9/volumes/kubernetes.io~projected/kube-api-access-c92f9:{mountpoint:/var/lib/kubelet/pods/d5210f23-6d88-413f-b0b5-c1b938713ca9/volumes/kubernetes.io~projected/kube-api-access-c92f9 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5210f23-6d88-413f-b0b5-c1b938713ca9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d5210f23-6d88-413f-b0b5-c1b938713ca9/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbd4817e-ed9f-4fea-858a-be4d25adcc1f/volumes/kubernetes.io~projected/kube-api-access-5qs7k:{mountpoint:/var/lib/kubelet/pods/dbd4817e-ed9f-4fea-858a-be4d25adcc1f/volumes/kubernetes.io~projected/kube-api-access-5qs7k major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de375da1-1b02-4796-ad49-2062bc5bc74b/volumes/kubernetes.io~projected/kube-api-access-8vq5n:{mountpoint:/var/lib/kubelet/pods/de375da1-1b02-4796-ad49-2062bc5bc74b/volumes/kubernetes.io~projected/kube-api-access-8vq5n major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0322b8e-5e35-4e52-b5dd-42f32573354f/volumes/kubernetes.io~projected/kube-api-access-x762b:{mountpoint:/var/lib/kubelet/pods/e0322b8e-5e35-4e52-b5dd-42f32573354f/volumes/kubernetes.io~projected/kube-api-access-x762b major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff/volumes/kubernetes.io~projected/kube-api-access-hqj6s:{mountpoint:/var/lib/kubelet/pods/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff/volumes/kubernetes.io~projected/kube-api-access-hqj6s major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3939775-883b-454d-9e3b-6949f6984dfb/volumes/kubernetes.io~projected/kube-api-access-5gwt5:{mountpoint:/var/lib/kubelet/pods/e3939775-883b-454d-9e3b-6949f6984dfb/volumes/kubernetes.io~projected/kube-api-access-5gwt5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a1808a-9488-4d49-8028-4860660c0c9d/volumes/kubernetes.io~projected/kube-api-access-79fkn:{mountpoint:/var/lib/kubelet/pods/e3a1808a-9488-4d49-8028-4860660c0c9d/volumes/kubernetes.io~projected/kube-api-access-79fkn major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a1808a-9488-4d49-8028-4860660c0c9d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e3a1808a-9488-4d49-8028-4860660c0c9d/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d/volumes/kubernetes.io~projected/kube-api-access-dskzm:{mountpoint:/var/lib/kubelet/pods/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d/volumes/kubernetes.io~projected/kube-api-access-dskzm major:0 minor:289 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a3e67928da0097c9212f58b62add16680b9971b799d8e3370d266022f0fdcce6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e78f61950650fc0363bcfb500be539b6a8e30c2e47047469e6121798eee36dfa/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/88cdd5f60d7780c6f23b467164e84e43d935cccc00cb45d96720b9a91e948c2e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/589ca95ea6d2bad2fc9717591037f5c5b57dcf2f768a661b3ae7592a4aa6fd14/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dfce2bb62d5d625be607c40dadeab87f888f586e413ec3ca2ce3ccb321590c30/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a144c33f500f0ef6e74489ec62dcb8b1d44d78c1220fa512b95f6043766e6361/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ed05058f27b858193534ddac035c483b3ad7f17ba9219d68bd2250c6a3620249/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4224f38b67ef64ede9a5dcaa1627cbc4bf9405a867c93188cf9837690327820a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/09ab844c6e902b4057c0c9b261ca0b644cd54b9ccf8606b017ed9e817febf86b/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/bc2a082f5fe93d8fdce3bfa64d3b04910b56931022e0f6566407e623194a6609/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/4b8cdc8e240c187b2825ee3bc21c61f2ac8ee82f1beb07bdb73bebdcab0d85ac/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/cae7b6e0472b3da23cc4cfd641bfdf1cd4657b695e29583523c320733bee8822/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8160c456cd4777e14c6444e107cab27b739d5ae2814579e002e4b166ace7b581/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/95e2b144e985a2d996953d85baf26f791047c37fe61a3c3cdec858be42df6a36/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d769e8cb0c7199d66c3a233900e0aeac3cce0c51252bee03faffdb7e26128096/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ea674c0f168a6441005e1a40a4791b4e484bb05eb1f74512ab890eded86a599c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2f804416351afb8f5ad3fced059b80f124a7b931a4471308682b22e437dfc0f5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/de5dd813eaf709de0e6871f0643f459822834babb3f8c202094e9332367ca125/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/479e83bed9b89346dc2dabf19eed847368a1a6fca7d7956c6eb0757ebe5ce687/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/aa54f8beec7febd04ae3b4c06c3fa5b675359bbaff9d8ecd47e967bc3d44ed5d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c4d0cf685d5679008fba55bb3ecd7fc6733e5d0a7dafc026ac0d065887748350/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/60ae5c03fdbbc95c5b4fdce96e6aca2e96fb417291b4a685cf5983534082c205/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/614bfc55ae5e21bd26ff4b6748fe66b8749fdce8085f59b834d943b46e9e4f69/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/b012d584d65e0a975da49981b2fe1df8e897f80e1025d8aac33195834bb27334/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/ce5105f95f7b687e54560132bf303d1e9526aa9c01b322821567fda1a4cca742/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/485ce73c6a4899113e440017c792517dee02e17e83b7c15470501638e45cd6fc/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/bf854c0985a61aebd9f13ff4a5093959439497fd814508c78c0506c6f38a532c/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/35bd7ee1d8137c0ec34c86cd1d8452bdc552ecd3f933f1d702cdfbfc72298506/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/5de22519b145df3d81be5a8570cf49daaff1fcacd64e877b6a1af87bf91c9013/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/a8cad1ca41a1c2df28aa0a7e8780be36421f741830ea3f43ce99f51b5003dfea/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/1a06a87f3f8f4b32a7188e9ef42aee93dda2a3ee5aaed9e744da7177b5e60810/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/444cde328a68b002c46f470710ff1547db049f1311c2ac0d4effc02cb4b1905d/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/b07d2ac1885842507200f649ec8edba71b737831d9adcf1706b3ececb64a3978/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/b2a9ee77e7381dae74373a867a8bc567e346a7477ac00cd494507723cbd7bbc9/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/91ca7d6f8ab994efdd3a429be93adae68686a52bb86c9732444d1a2d94a24120/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/8999eb5b83ecce4217166817ecc8dfd88d1380f09788d71f2d22a68bb64467f6/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/6ce8924b6b7f88afa06d1bd00058ba91fbb00df59b94deecae3991d15d05868a/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/22650b267a1df61e39614da02ee3bacb1a5680af58b7fed65aa4ac15f2be168b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/697c55888bea3f6440eba54f2b9c25fc7c6e041dfe7b3c3fae5f6cf37d901129/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/ba41b43fde66248d371e77085095e0a4de803b7464a1530be6e2a01c8acb52ee/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/50fdc0152f991b4d247c388ff90b4028a661660e4f1c88fe837fc8dfd9e79658/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/007c33534860d43e6ea9c577a837ab7b9fcdd66e23b63076e95a42db0e85d4f5/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/448d743472c1c5be83423042e5f859ddf7b3231e7ebd988bd2ffd30774a5da75/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/00ba6e2d3ff0e31d65f01cc2eb28f565da58c23e2551cf0460a51ab9efcfa317/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/826262b2eeec3e332864ae41de0d9a12eacf8b7d36b872a61cbbd1b793a70cee/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/b05d4f9f2d241e149dae28b577b028ad39dbbec76a14231459a36b06b45caf0e/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/ca43cac84537b518f1819e403d9fa41e50b784ac584e049c2e07e95edad8e165/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/32d7ce142a94af8f3d1b401e65644e9e0483a916690c3a5539056acfc988ae99/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/98fa5445d0ebdb661dbb7743ca972d2adae11efbcc3b80b579a9a14f8492aa1c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f11ed9c9163e3a441b15a35ce72b66dbb337d8ab2f71f952c0cae20836ac2f7e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6aa3089d2f8ed82440dddec3c38274eee73a48a15e7b57daf18bc243dab5dc63/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/94ff4759e7775f5fe8878fd5ecc48b26e56c73a0a5af90da83390059cee9e72b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/215f4f07a6779c30d5b9316269b2aa8832bf670180268c93ba778a9afc2876db/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f9c75687f1d1327508efba1ca2806580c5d671bf26525cbe986a4d746e803a50/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a3723cc53da12f95d5bacbac26670931002f500c0e4a46ef3f201801a5d94f8d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ff827e206404862aa23f13b6ef92c977c2a4f11033bf55e8f26a012688b60e53/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.847672 4898 manager.go:217] Machine: {Timestamp:2026-11-03 22:22:36.846675329 +0000 UTC m=+0.059444374 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3207330 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b87875cc38baeb20897ee2bb7ffef SystemUUID:ec2b8787-5cc3-8bae-b208-97ee2bb7ffef BootID:352bf077-e266-41ff-9fcc-9d5a91a40049 Filesystems:[{Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a1808a-9488-4d49-8028-4860660c0c9d/volumes/kubernetes.io~projected/kube-api-access-79fkn DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/64ebd614-43bd-479e-8e95-c33b2c93361c/volumes/kubernetes.io~projected/kube-api-access-g6xjb DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06b3993b-eaa8-4649-9c38-2b9bb7a22498/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1ce9e0ec2670160ecf38165c74868eed7e49647387b153a664244b8a8fd82dd/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7058c42487b134a7e407662f1367cf2426cca7a8b7a3cbf07000fcb94d98e7dc/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68e1c673-629b-4840-9f7a-aa64777cf32a/volumes/kubernetes.io~projected/kube-api-access-j5jpx DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~projected/kube-api-access-g5pmx DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3939775-883b-454d-9e3b-6949f6984dfb/volumes/kubernetes.io~projected/kube-api-access-5gwt5 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cc81f17-d52c-4edb-9f33-de2f37c80b54/volumes/kubernetes.io~projected/kube-api-access-qqwpg DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a374ff-7146-47f5-bcbd-6c4dc710d580/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/add89f53-e47e-41de-8d0d-6fc06d826d29/volumes/kubernetes.io~projected/kube-api-access-dstxj DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e43502be7dc286327ba3930be1ff7900403ba713d6f0d2a5b50b2f2592c69a36/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f86ef3f4fa9f6d1db1a27cd235dfd4bcc78a9105f6e3862064dfc0493b834c8f/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34/volumes/kubernetes.io~projected/kube-api-access-kpnfb DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53db53896889d659be7345968316c874f3162d7f37c7186f83ee26b6b7a56b2c/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a1808a-9488-4d49-8028-4860660c0c9d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a5b40b6b115159914681045eafc12f661fa66018aa7af33537aa13109a3b71f/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de375da1-1b02-4796-ad49-2062bc5bc74b/volumes/kubernetes.io~projected/kube-api-access-8vq5n DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d/volumes/kubernetes.io~projected/kube-api-access-dskzm DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volumes/kubernetes.io~projected/kube-api-access-dsqkb DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c95c406-8766-4a77-9639-513f2f37c1d3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60c55366c10392824707b262927dd6545f2141c7ca0ee2ae3b98efef3bc1d6fe/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f366bb5ef5917b9fbc4d75946c8250753bac840431368aef2ef57a9768682a5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8bfc9f4803efea7ff26427281671784a21595fd246c625a963545f2036c2508/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34/volumes/kubernetes.io~projected/kube-api-access-j9qrs DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0322b8e-5e35-4e52-b5dd-42f32573354f/volumes/kubernetes.io~projected/kube-api-access-x762b DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff/volumes/kubernetes.io~projected/kube-api-access-hqj6s DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f23b7eb-3f0e-4215-a42c-c20c976900dc/volumes/kubernetes.io~projected/kube-api-access-cgpc9 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79e643b9bcc8e215e963ff27d4255c28ec8f908a97d1b541ba261eac021b9907/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b81844cd89060dd454ac71b792cc01cfdf446a2e5ff01e9dc4fb444d37946637/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65f4d5dd-a512-425a-91b9-de7749f091c6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1ad8cc30fd5c61fd78a184c67e09459ad82764d7840fe2b211856aaa07994e7/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d7234dd-7b73-427d-b0e1-afd778bd2028/volumes/kubernetes.io~projected/kube-api-access-fxd8q DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~projected/kube-api-access-bzvnh DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b98dc03494e8dd3e1b4a96275c3281d260331648afb07927d568144c48df44b/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6795c510085a438fb9965116ccfadbd399a6ade6b86b58ac66a031a8aa14aecf/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/458febd7-7dcd-4ef1-9b8b-27037557b753/volumes/kubernetes.io~projected/kube-api-access-9wjrh DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c95c406-8766-4a77-9639-513f2f37c1d3/volumes/kubernetes.io~projected/kube-api-access-sbfgm DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60b13fc0432176f1ed91db525e823ea8e00d1c18f237de750d591eebc3058a62/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cc6466a6ca067e1ac7162761d63058ae27fd4fd442195ff24253063ba9e40dc/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ee286eb53183ff3d40245d8fbb2be7ee833ac2bf8697cf5d45d12d63789033b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5210f23-6d88-413f-b0b5-c1b938713ca9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbd4817e-ed9f-4fea-858a-be4d25adcc1f/volumes/kubernetes.io~projected/kube-api-access-5qs7k DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9b5b425-585c-4fc9-bc11-2510c7035c70/volumes/kubernetes.io~projected/kube-api-access-sx8xp DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/478ac8d747bd642b9bfb5ed1136dbc4c3b8ed747122f7dbeb5923b0d57f05ac0/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes/kubernetes.io~projected/kube-api-access-ghp7z DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~projected/kube-api-access-knc9r DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65f4d5dd-a512-425a-91b9-de7749f091c6/volumes/kubernetes.io~projected/kube-api-access-4l29j DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23bfb68550386e57bbfef9fc5811d99c33613e35a85be46da473af252806b87c/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5210f23-6d88-413f-b0b5-c1b938713ca9/volumes/kubernetes.io~projected/kube-api-access-c92f9 DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acf959bc9242b1eaac3a5d71f53c19eaffa6993649deb71f5ddbae7427c0d433/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3714e210-a948-4ba6-a174-0fc8eb8e06db/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700b9d15-342a-46d1-b94e-6b44aae2d9b5/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/724a681a-c911-439e-aa2c-904081578713/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b6fe9b11-1b54-4871-9236-dae9154b43fb/volumes/kubernetes.io~projected/kube-api-access-4fh74 DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35ea235a-c0ba-44b9-909f-e3dd472a6c36/volumes/kubernetes.io~projected/kube-api-access-cgqnx DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60be4f8c-7051-444d-ad87-7e7895c94542/volumes/kubernetes.io~projected/kube-api-access-hpk4w DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:23bfb68550386e5 MacAddress:de:14:fb:c3:af:bc Speed:10000 Mtu:8901} {Name:2a3cfb6a46c5b5d MacAddress:32:69:f4:46:75:53 Speed:10000 Mtu:8901} {Name:2a5b40b6b115159 MacAddress:52:9c:5d:eb:77:8a Speed:10000 Mtu:8901} {Name:2b98dc03494e8dd MacAddress:c2:b5:4d:9f:5d:50 Speed:10000 Mtu:8901} {Name:2cc6466a6ca067e MacAddress:26:b9:ff:19:03:f1 Speed:10000 Mtu:8901} {Name:478ac8d747bd642 MacAddress:c6:ff:4b:53:58:9e Speed:10000 Mtu:8901} {Name:53db53896889d65 MacAddress:ca:7d:0c:a1:8d:90 Speed:10000 Mtu:8901} {Name:6795c510085a438 MacAddress:1a:6b:99:4b:02:1c Speed:10000 Mtu:8901} {Name:7058c42487b134a MacAddress:72:36:89:f2:19:56 Speed:10000 Mtu:8901} {Name:79e643b9bcc8e21 MacAddress:1a:cf:5d:48:28:a1 Speed:10000 Mtu:8901} {Name:a1ad8cc30fd5c61 MacAddress:06:0f:5f:a7:ac:ac Speed:10000 Mtu:8901} {Name:acf959bc9242b1e MacAddress:ba:1e:17:91:49:02 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f7:ad:62:71 Speed:0 Mtu:9001} {Name:br-int MacAddress:02:c5:1b:c5:5c:c0 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:f7:ad:62:71 Speed:0 Mtu:9001} {Name:f86ef3f4fa9f6d1 MacAddress:ce:4e:d5:4d:8a:25 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:40:0a:e7:9e:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848061 4898 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848115 4898 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848372 4898 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848389 4898 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-173.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848508 4898 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848516 4898 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848527 4898 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848547 4898 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848566 4898 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848703 4898 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:22:36.848821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848828 4898 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:22:36.849385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848870 4898 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:22:36.849385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848884 4898 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:22:36.849385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848893 4898 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:22:36.849385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848902 4898 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:22:36.849385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.848910 4898 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:22:36.849534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.849521 4898 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:22:36.849566 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.849534 4898 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:36.850015 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.849992 4898 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:22:36.852728 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.852713 4898 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:22:36.852728 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.852728 4898 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:36.852854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.852843 4898 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:22:36.853059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853048 4898 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:22:36.853095 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853077 4898 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:22:36.853207 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853194 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:22:36.853236 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853217 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:22:36.853236 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853225 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:22:36.853286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853242 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:22:36.853286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853248 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:22:36.853286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853254 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:22:36.853286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853284 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:22:36.853401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853292 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:22:36.853401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853302 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:22:36.853401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853310 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:22:36.853401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853344 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:22:36.853401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853358 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:22:36.853401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853391 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:22:36.853401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.853399 4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:22:36.857599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.857582 4898 server.go:1263] "Started kubelet" Nov 03 22:22:36.857697 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.857634 4898 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:22:36.858107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.858030 4898 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:22:36.858186 ip-10-0-139-173 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:22:36.858461 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.858205 4898 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:22:36.858582 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.858551 4898 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:22:36.861034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.861004 4898 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:22:36.861135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.861075 4898 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:22:36.863397 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.863303 4898 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:22:36.863397 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.863402 4898 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:22:36.863574 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.863521 4898 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:57 +0000 UTC" deadline="2028-08-06 16:36:36.188273848 +0000 UTC" Nov 03 22:22:36.863574 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.863556 4898 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15402h13m59.324721364s" Nov 03 22:22:36.865162 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:36.863898 4898 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:22:36.865265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.865210 4898 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:22:36.865596 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.865580 4898 factory.go:55] Registering systemd factory Nov 03 22:22:36.865784 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.865601 4898 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:22:36.865957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.865945 4898 factory.go:153] Registering CRI-O factory Nov 03 22:22:36.866031 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.865961 4898 factory.go:223] Registration of the crio container factory successfully Nov 03 22:22:36.866100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.866044 4898 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:22:36.866100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.866064 4898 factory.go:103] Registering Raw factory Nov 03 22:22:36.866100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.866083 4898 manager.go:1199] Started watching for new ooms in manager Nov 03 22:22:36.866482 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.866468 4898 manager.go:319] Starting recovery of all containers Nov 03 22:22:36.868667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.868645 4898 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:22:36.869604 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869566 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="458febd7-7dcd-4ef1-9b8b-27037557b753" volumeName="kubernetes.io/projected/458febd7-7dcd-4ef1-9b8b-27037557b753-kube-api-access-9wjrh" seLinuxMountContext="" Nov 03 22:22:36.869695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869610 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34" volumeName="kubernetes.io/secret/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-serving-cert" seLinuxMountContext="" Nov 03 22:22:36.869695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869626 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724a681a-c911-439e-aa2c-904081578713" volumeName="kubernetes.io/projected/724a681a-c911-439e-aa2c-904081578713-kube-api-access-dsqkb" seLinuxMountContext="" Nov 03 22:22:36.869695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869640 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cc81f17-d52c-4edb-9f33-de2f37c80b54" volumeName="kubernetes.io/empty-dir/7cc81f17-d52c-4edb-9f33-de2f37c80b54-data-volume" seLinuxMountContext="" Nov 03 22:22:36.869695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869654 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f621d2a-1926-4044-80bb-d64a0574e356" volumeName="kubernetes.io/configmap/7f621d2a-1926-4044-80bb-d64a0574e356-nginx-conf" seLinuxMountContext="" Nov 03 22:22:36.869695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869670 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5df88-99b0-4026-8ae2-b71752948379" volumeName="kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:22:36.869695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869682 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3714e210-a948-4ba6-a174-0fc8eb8e06db" volumeName="kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-tuned" seLinuxMountContext="" Nov 03 22:22:36.869695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869693 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34" volumeName="kubernetes.io/configmap/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-config" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869712 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cc81f17-d52c-4edb-9f33-de2f37c80b54" volumeName="kubernetes.io/projected/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-api-access-qqwpg" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869727 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5210f23-6d88-413f-b0b5-c1b938713ca9" volumeName="kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869739 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34" volumeName="kubernetes.io/projected/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-kube-api-access-kpnfb" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869752 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65f4d5dd-a512-425a-91b9-de7749f091c6" volumeName="kubernetes.io/secret/65f4d5dd-a512-425a-91b9-de7749f091c6-serving-cert" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869772 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724a681a-c911-439e-aa2c-904081578713" volumeName="kubernetes.io/secret/724a681a-c911-439e-aa2c-904081578713-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869810 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0322b8e-5e35-4e52-b5dd-42f32573354f" volumeName="kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869825 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" volumeName="kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869837 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e1c673-629b-4840-9f7a-aa64777cf32a" volumeName="kubernetes.io/projected/68e1c673-629b-4840-9f7a-aa64777cf32a-kube-api-access-j5jpx" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869865 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a1808a-9488-4d49-8028-4860660c0c9d" volumeName="kubernetes.io/secret/e3a1808a-9488-4d49-8028-4860660c0c9d-serving-cert" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869878 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f23b7eb-3f0e-4215-a42c-c20c976900dc" volumeName="kubernetes.io/projected/1f23b7eb-3f0e-4215-a42c-c20c976900dc-kube-api-access-cgpc9" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869891 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" volumeName="kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869903 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5210f23-6d88-413f-b0b5-c1b938713ca9" volumeName="kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-snapshots" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869914 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3714e210-a948-4ba6-a174-0fc8eb8e06db" volumeName="kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-tmp" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869927 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5b425-585c-4fc9-bc11-2510c7035c70" volumeName="kubernetes.io/projected/a9b5b425-585c-4fc9-bc11-2510c7035c70-kube-api-access-sx8xp" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869951 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5210f23-6d88-413f-b0b5-c1b938713ca9" volumeName="kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-service-ca-bundle" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869964 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a3e17d-abf7-4d4b-afe0-fd028d2caeff" volumeName="kubernetes.io/projected/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff-kube-api-access-hqj6s" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869976 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d7234dd-7b73-427d-b0e1-afd778bd2028" volumeName="kubernetes.io/empty-dir/1d7234dd-7b73-427d-b0e1-afd778bd2028-tmp-dir" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.869990 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f23b7eb-3f0e-4215-a42c-c20c976900dc" volumeName="kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870003 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" volumeName="kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870014 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbd4817e-ed9f-4fea-858a-be4d25adcc1f" volumeName="kubernetes.io/configmap/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-serviceca" seLinuxMountContext="" Nov 03 22:22:36.870042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870032 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" volumeName="kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870057 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65f4d5dd-a512-425a-91b9-de7749f091c6" volumeName="kubernetes.io/configmap/65f4d5dd-a512-425a-91b9-de7749f091c6-config" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870071 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e1c673-629b-4840-9f7a-aa64777cf32a" volumeName="kubernetes.io/empty-dir/68e1c673-629b-4840-9f7a-aa64777cf32a-tmp-dir" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870082 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700b9d15-342a-46d1-b94e-6b44aae2d9b5" volumeName="kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-stats-auth" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870094 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" volumeName="kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870107 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e1c673-629b-4840-9f7a-aa64777cf32a" volumeName="kubernetes.io/configmap/68e1c673-629b-4840-9f7a-aa64777cf32a-config-volume" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870119 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6fe9b11-1b54-4871-9236-dae9154b43fb" volumeName="kubernetes.io/projected/b6fe9b11-1b54-4871-9236-dae9154b43fb-kube-api-access-4fh74" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870131 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0322b8e-5e35-4e52-b5dd-42f32573354f" volumeName="kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870144 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5b425-585c-4fc9-bc11-2510c7035c70" volumeName="kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-cni-binary-copy" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870155 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64ebd614-43bd-479e-8e95-c33b2c93361c" volumeName="kubernetes.io/configmap/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-config" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870167 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700b9d15-342a-46d1-b94e-6b44aae2d9b5" volumeName="kubernetes.io/projected/700b9d15-342a-46d1-b94e-6b44aae2d9b5-kube-api-access-g5pmx" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870179 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724a681a-c911-439e-aa2c-904081578713" volumeName="kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-env-overrides" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870191 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a374ff-7146-47f5-bcbd-6c4dc710d580" volumeName="kubernetes.io/secret/c1a374ff-7146-47f5-bcbd-6c4dc710d580-agent-certs" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870204 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5210f23-6d88-413f-b0b5-c1b938713ca9" volumeName="kubernetes.io/projected/d5210f23-6d88-413f-b0b5-c1b938713ca9-kube-api-access-c92f9" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870227 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de375da1-1b02-4796-ad49-2062bc5bc74b" volumeName="kubernetes.io/projected/de375da1-1b02-4796-ad49-2062bc5bc74b-kube-api-access-8vq5n" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870239 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3939775-883b-454d-9e3b-6949f6984dfb" volumeName="kubernetes.io/projected/e3939775-883b-454d-9e3b-6949f6984dfb-kube-api-access-5gwt5" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870251 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5df88-99b0-4026-8ae2-b71752948379" volumeName="kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870263 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d7234dd-7b73-427d-b0e1-afd778bd2028" volumeName="kubernetes.io/projected/1d7234dd-7b73-427d-b0e1-afd778bd2028-kube-api-access-fxd8q" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870274 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" volumeName="kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870287 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60be4f8c-7051-444d-ad87-7e7895c94542" volumeName="kubernetes.io/projected/60be4f8c-7051-444d-ad87-7e7895c94542-kube-api-access-hpk4w" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870299 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724a681a-c911-439e-aa2c-904081578713" volumeName="kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-config" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870311 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a1808a-9488-4d49-8028-4860660c0c9d" volumeName="kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-trusted-ca" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870323 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06b3993b-eaa8-4649-9c38-2b9bb7a22498" volumeName="kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870336 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3714e210-a948-4ba6-a174-0fc8eb8e06db" volumeName="kubernetes.io/projected/3714e210-a948-4ba6-a174-0fc8eb8e06db-kube-api-access-bzvnh" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870348 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a374ff-7146-47f5-bcbd-6c4dc710d580" volumeName="kubernetes.io/configmap/c1a374ff-7146-47f5-bcbd-6c4dc710d580-konnectivity-ca" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870371 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5210f23-6d88-413f-b0b5-c1b938713ca9" volumeName="kubernetes.io/secret/d5210f23-6d88-413f-b0b5-c1b938713ca9-serving-cert" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870384 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0322b8e-5e35-4e52-b5dd-42f32573354f" volumeName="kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870398 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5df88-99b0-4026-8ae2-b71752948379" volumeName="kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870422 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5df88-99b0-4026-8ae2-b71752948379" volumeName="kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870435 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ea235a-c0ba-44b9-909f-e3dd472a6c36" volumeName="kubernetes.io/projected/35ea235a-c0ba-44b9-909f-e3dd472a6c36-kube-api-access-cgqnx" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870448 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64ebd614-43bd-479e-8e95-c33b2c93361c" volumeName="kubernetes.io/projected/64ebd614-43bd-479e-8e95-c33b2c93361c-kube-api-access-g6xjb" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870460 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65f4d5dd-a512-425a-91b9-de7749f091c6" volumeName="kubernetes.io/projected/65f4d5dd-a512-425a-91b9-de7749f091c6-kube-api-access-4l29j" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870472 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724a681a-c911-439e-aa2c-904081578713" volumeName="kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870484 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c95c406-8766-4a77-9639-513f2f37c1d3" volumeName="kubernetes.io/projected/7c95c406-8766-4a77-9639-513f2f37c1d3-kube-api-access-sbfgm" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870497 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5b425-585c-4fc9-bc11-2510c7035c70" volumeName="kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-daemon-config" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870508 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5df88-99b0-4026-8ae2-b71752948379" volumeName="kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870521 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60be4f8c-7051-444d-ad87-7e7895c94542" volumeName="kubernetes.io/configmap/60be4f8c-7051-444d-ad87-7e7895c94542-telemetry-config" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870534 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c95c406-8766-4a77-9639-513f2f37c1d3" volumeName="kubernetes.io/secret/7c95c406-8766-4a77-9639-513f2f37c1d3-cert" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870545 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbd4817e-ed9f-4fea-858a-be4d25adcc1f" volumeName="kubernetes.io/projected/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-kube-api-access-5qs7k" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870559 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0322b8e-5e35-4e52-b5dd-42f32573354f" volumeName="kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870571 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a1808a-9488-4d49-8028-4860660c0c9d" volumeName="kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-config" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870584 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a1808a-9488-4d49-8028-4860660c0c9d" volumeName="kubernetes.io/projected/e3a1808a-9488-4d49-8028-4860660c0c9d-kube-api-access-79fkn" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870595 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8d3725a-cbd4-4ef3-a433-2010c83a5f3d" volumeName="kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870607 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34" volumeName="kubernetes.io/projected/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34-kube-api-access-j9qrs" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870619 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="458febd7-7dcd-4ef1-9b8b-27037557b753" volumeName="kubernetes.io/configmap/458febd7-7dcd-4ef1-9b8b-27037557b753-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870632 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700b9d15-342a-46d1-b94e-6b44aae2d9b5" volumeName="kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-default-certificate" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870643 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5210f23-6d88-413f-b0b5-c1b938713ca9" volumeName="kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-tmp" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870654 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5df88-99b0-4026-8ae2-b71752948379" volumeName="kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870666 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5df88-99b0-4026-8ae2-b71752948379" volumeName="kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870678 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f23b7eb-3f0e-4215-a42c-c20c976900dc" volumeName="kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-binary-copy" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870690 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add89f53-e47e-41de-8d0d-6fc06d826d29" volumeName="kubernetes.io/projected/add89f53-e47e-41de-8d0d-6fc06d826d29-kube-api-access-dstxj" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870702 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f23b7eb-3f0e-4215-a42c-c20c976900dc" volumeName="kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870715 4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" volumeName="kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates" seLinuxMountContext="" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870727 4898 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:22:36.870988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.870735 4898 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:22:36.887461 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.887437 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/1.log" Nov 03 22:22:36.887570 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.887438 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/1.log" Nov 03 22:22:36.893961 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.893823 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:22:36.894870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.894847 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:22:36.896666 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.896637 4898 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:22:36.898241 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.898218 4898 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:22:36.898241 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.898243 4898 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:22:36.898385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.898262 4898 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:22:36.898385 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:36.898293 4898 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:22:36.915598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.915556 4898 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="7375d0346f8aa57c4088ec3f28b66e7bd849041ce33efc9773b4267e61970205" exitCode=0 Nov 03 22:22:36.915598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.915587 4898 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="bdbbf881f4defa5095fa920522bccad0451a764d7724b11b19cdf4fddccad9ce" exitCode=0 Nov 03 22:22:36.915598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.915594 4898 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="c9ab69928c2b0652d7a9a2069c8a1e5c3462a1308f37f5946dc6a4acfc15efb4" exitCode=0 Nov 03 22:22:36.915598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.915601 4898 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="5f8afe677d7e40921f2ac1af605b4db4c7db1369bd9c9eefef4514c4c1919ddf" exitCode=0 Nov 03 22:22:36.915880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.915609 4898 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="c8fee35c220045e2c029953e76442373b6be086736fc41d937ba045c205715d2" exitCode=0 Nov 03 22:22:36.915880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.915614 4898 generic.go:356] "Generic (PLEG): container finished" podID="1f23b7eb-3f0e-4215-a42c-c20c976900dc" containerID="c0803a16dab7a8b3f13ff57348fdea10f469c263a365af135256613ee1d8f42e" exitCode=0 Nov 03 22:22:36.930811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.930773 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/1.log" Nov 03 22:22:36.930925 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.930823 4898 generic.go:356] "Generic (PLEG): container finished" podID="e3a1808a-9488-4d49-8028-4860660c0c9d" containerID="fbf7747233d82ae2ff25e3a729e1ddf99197402b0e6cff13442a2475f1cc27bf" exitCode=255 Nov 03 22:22:36.936859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.936837 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:22:36.937171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.937143 4898 generic.go:356] "Generic (PLEG): container finished" podID="724a681a-c911-439e-aa2c-904081578713" containerID="ffc553080bd220a81fafed5b87ebe6866d10f4aaf755d191e9f6c9e0e6f26216" exitCode=1 Nov 03 22:22:36.949009 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.948967 4898 generic.go:356] "Generic (PLEG): container finished" podID="d4c43c551f0a2ae66c9cba55c176ecea" containerID="e8e30862984064063b791adaa20d337fad7f94bc0ca45085998069f9dfe09895" exitCode=0 Nov 03 22:22:36.963973 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:36.963950 4898 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:22:36.967761 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.967742 4898 manager.go:324] Recovery completed Nov 03 22:22:36.985540 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.985484 4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:36.986661 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.986645 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:36.986746 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.986679 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:36.986746 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.986694 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:36.989239 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989223 4898 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:22:36.989239 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989237 4898 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:22:36.989390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989240 4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:36.989390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989257 4898 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:22:36.989488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989409 4898 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:22:36.989488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989425 4898 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:22:36.989488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989449 4898 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:22:36.989488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989458 4898 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:22:36.989488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989466 4898 policy_none.go:50] "Start" Nov 03 22:22:36.989488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989478 4898 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:22:36.989758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989491 4898 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:22:36.989758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989667 4898 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:22:36.989758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989678 4898 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:22:36.989758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.989687 4898 policy_none.go:44] "Start" Nov 03 22:22:36.990125 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.990105 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:36.990184 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.990142 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:36.990184 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.990155 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:36.994277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.994251 4898 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:22:36.997048 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997034 4898 manager.go:341] "Starting Device Plugin manager" Nov 03 22:22:36.997116 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:36.997060 4898 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:22:36.997116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997070 4898 server.go:87] "Starting device plugin registration server" Nov 03 22:22:36.997337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997326 4898 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:22:36.997370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997338 4898 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:22:36.997470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997448 4898 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:22:36.997592 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997562 4898 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:22:36.997665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997601 4898 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:22:36.997665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997608 4898 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:22:36.997770 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.997666 4898 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:22:36.99759055Z","UUID":"3069dfc4-7402-4f99-b18f-f668d6b7305e","Handler":null,"Name":"","Endpoint":""} Nov 03 22:22:36.998390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.998354 4898 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal"] Nov 03 22:22:36.998848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:36.998775 4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:37.000441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.000416 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:37.000579 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.000551 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:37.000664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.000581 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:37.001063 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:37.001025 4898 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:22:37.001225 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.001211 4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:37.001385 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:37.001368 4898 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" Nov 03 22:22:37.001457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.001439 4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:37.002641 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.002625 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:37.002721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.002673 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:37.002721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.002700 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:37.002721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.002708 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:37.002882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.002732 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:37.002882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.002744 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:37.002983 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:37.002961 4898 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.003320 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.003302 4898 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:22:37.003413 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.003327 4898 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:22:37.003610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.003577 4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:37.003935 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.003892 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"d4c43c551f0a2ae66c9cba55c176ecea","Type":"ContainerStarted","Data":"edbf084cb9d3618769e229e3c3e6925e17d0a5cc0851136c5f896577b47f6ee5"} Nov 03 22:22:37.004029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.003950 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"d4c43c551f0a2ae66c9cba55c176ecea","Type":"ContainerDied","Data":"e8e30862984064063b791adaa20d337fad7f94bc0ca45085998069f9dfe09895"} Nov 03 22:22:37.004029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.003973 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"d4c43c551f0a2ae66c9cba55c176ecea","Type":"ContainerStarted","Data":"03813b3d5221d2d8162fde28ce471b5c60326c746d8987e44da9547ba73c3a1e"} Nov 03 22:22:37.004029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.004010 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"3dade66e2d9f1769c6a8531aeeaf63ad","Type":"ContainerStarted","Data":"3a1cbad6f8d0d46f81c985d416b74697a3261ab54595c05ead23a1afa6eea666"} Nov 03 22:22:37.004191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.004031 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" event={"ID":"3dade66e2d9f1769c6a8531aeeaf63ad","Type":"ContainerStarted","Data":"c7899ba7fbfa904c976aa4bbc16f2b43eb8a99cb1d23f1febd2027ca9e5b1d70"} Nov 03 22:22:37.007382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.004746 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:37.007382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.004777 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:37.007382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.004812 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:37.007382 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:37.004953 4898 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-173.us-west-2.compute.internal\" not found" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.064421 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.064396 4898 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:22:37.065633 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.065598 4898 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:37.065811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.065775 4898 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:37.066585 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.066564 4898 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:37.097485 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.097453 4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:37.098601 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.098583 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:37.098655 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.098622 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:37.098655 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.098637 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:37.098734 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.098683 4898 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.117155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.117125 4898 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:37.165569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.165541 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3dade66e2d9f1769c6a8531aeeaf63ad-config\") pod \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"3dade66e2d9f1769c6a8531aeeaf63ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.165723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.165574 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.165723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.165550 4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.165723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.165596 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.209056 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.209027 4898 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.209177 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.209094 4898 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.209669 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.209652 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeNotReady" Nov 03 22:22:37.209730 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.209688 4898 setters.go:546] "Node became not ready" node="ip-10-0-139-173.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T22:22:37Z","lastTransitionTime":"2026-11-03T22:22:37Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 22:22:37.228464 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.228435 4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:37.228651 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:37.228482 4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.228651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.228496 4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.233104 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.233079 4898 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-173.us-west-2.compute.internal" is owned by different node (oldNodeID="56b76859-427b-4f13-b9bc-578a77a13615", newNodeID=""), cleaning up... Nov 03 22:22:37.266088 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.266028 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3dade66e2d9f1769c6a8531aeeaf63ad-config\") pod \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"3dade66e2d9f1769c6a8531aeeaf63ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.266173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.266100 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3dade66e2d9f1769c6a8531aeeaf63ad-config\") pod \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"3dade66e2d9f1769c6a8531aeeaf63ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.266173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.266120 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.266173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.266149 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.266266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.266186 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.266266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.266212 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4c43c551f0a2ae66c9cba55c176ecea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" (UID: \"d4c43c551f0a2ae66c9cba55c176ecea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.293572 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.293548 4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:37.293677 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:37.293593 4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.850116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.850083 4898 apiserver.go:52] "Watching apiserver" Nov 03 22:22:37.971760 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.971690 4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:37.971913 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:37.971847 4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:38.004931 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.004903 4898 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:22:38.005901 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.005869 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d9b64b6dc-xqdzm","openshift-multus/multus-pjjqw","openshift-dns/dns-default-jmjbp","openshift-insights/insights-operator-7c4986d449-z5lhd","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw","openshift-image-registry/image-registry-58d49cb494-tnd66","openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg","openshift-multus/network-metrics-daemon-rcrd8","kube-system/kas-connection-checker-685895c7c9-6ddbh","openshift-dns/node-resolver-fqr6l","openshift-network-diagnostics/network-check-target-mzwbr","openshift-network-operator/iptables-alerter-qsnsn","kube-system/global-pull-secret-syncer-c9btg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92","openshift-ingress/router-default-74d647c79-nsnqv","kube-system/kas-connection-checker-685895c7c9-xrnfv","openshift-cluster-node-tuning-operator/tuned-zg9h5","openshift-console-operator/console-operator-7cd56b8dbc-thjpw","openshift-ingress-canary/ingress-canary-pqv9b","kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-jkxgz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5","openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb","kube-system/konnectivity-agent-pvpqf","openshift-image-registry/node-ca-2g542","openshift-insights/insights-runtime-extractor-r8md2","openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6","openshift-service-ca/service-ca-5fffc97bdf-m5pmk","kube-system/kas-connection-checker-685895c7c9-hdlkm","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs","openshift-insights/periodic-gathering-fnl7t-pdbr9","openshift-multus/multus-additional-cni-plugins-jqclr","openshift-ovn-kubernetes/ovnkube-node-rhx44","openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j"] Nov 03 22:22:38.009614 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.009574 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.010960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.010931 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.012388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.012369 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.012734 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.012720 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.013242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.013210 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:38.016369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.016353 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:38.018518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.018501 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.019844 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.019826 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.020581 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.020568 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.024293 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.024274 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" Nov 03 22:22:38.026532 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.026514 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" Nov 03 22:22:38.028026 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.028012 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.035997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.035967 4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:38.036063 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.036028 4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:38.037182 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.037165 4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:38.037253 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.037204 4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-173.us-west-2.compute.internal" Nov 03 22:22:38.061376 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.061354 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.061376 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.061367 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.065808 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.065773 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.069180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.069148 4898 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:22:38.071499 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071471 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l29j\" (UniqueName: \"kubernetes.io/projected/65f4d5dd-a512-425a-91b9-de7749f091c6-kube-api-access-4l29j\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:38.071610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071512 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghp7z\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.071610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071587 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:22:38.071610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071609 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:22:38.071900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071715 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xjb\" (UniqueName: \"kubernetes.io/projected/64ebd614-43bd-479e-8e95-c33b2c93361c-kube-api-access-g6xjb\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.071900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071748 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-tuned\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.071900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071829 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:22:38.071900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071866 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-system-cni-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071913 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a374ff-7146-47f5-bcbd-6c4dc710d580-agent-certs\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071924 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071942 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e1c673-629b-4840-9f7a-aa64777cf32a-tmp-dir\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071967 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-tuned\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.071978 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-netns\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072013 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vq5n\" (UniqueName: \"kubernetes.io/projected/de375da1-1b02-4796-ad49-2062bc5bc74b-kube-api-access-8vq5n\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072057 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e1c673-629b-4840-9f7a-aa64777cf32a-tmp-dir\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.072099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072098 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072112 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gg966\"" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072163 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072224 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-slash\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072233 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072264 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-systemd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072287 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h9m4r\"" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072298 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072329 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.072430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072381 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072437 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x6gxc\"" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072503 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072556 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-bin\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072581 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xd6w\" (UniqueName: \"kubernetes.io/projected/28365524-7763-4515-93fd-379ea89ffd5e-kube-api-access-8xd6w\") pod \"migrator-66df5d496b-72zn6\" (UID: \"28365524-7763-4515-93fd-379ea89ffd5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072590 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072626 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072629 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pq9zt\"" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072659 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-lib-modules\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072687 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-tmp\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072707 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072730 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dd97f\"" Nov 03 22:22:38.072740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072733 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072804 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-t48dr\"" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072818 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3714e210-a948-4ba6-a174-0fc8eb8e06db-tmp\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072842 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072871 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8l7wg\"" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072874 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072910 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxd8q\" (UniqueName: \"kubernetes.io/projected/1d7234dd-7b73-427d-b0e1-afd778bd2028-kube-api-access-fxd8q\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072935 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072951 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqkb\" (UniqueName: \"kubernetes.io/projected/724a681a-c911-439e-aa2c-904081578713-kube-api-access-dsqkb\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072967 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.072985 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073111 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwt5\" (UniqueName: \"kubernetes.io/projected/e3939775-883b-454d-9e3b-6949f6984dfb-kube-api-access-5gwt5\") pod \"kas-connection-checker-685895c7c9-6ddbh\" (UID: \"e3939775-883b-454d-9e3b-6949f6984dfb\") " pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" Nov 03 22:22:38.073181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073149 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60be4f8c-7051-444d-ad87-7e7895c94542-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073309 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnfb\" (UniqueName: \"kubernetes.io/projected/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-kube-api-access-kpnfb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073331 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-modprobe-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073346 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073362 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073384 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-socket-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073432 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysconfig\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073449 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-systemd\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073484 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073503 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pw8n\" (UniqueName: \"kubernetes.io/projected/2ec50892-7849-4b30-934a-c9983238c501-kube-api-access-6pw8n\") pod \"migrator-66df5d496b-m2dcb\" (UID: \"2ec50892-7849-4b30-934a-c9983238c501\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073521 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724a681a-c911-439e-aa2c-904081578713-ovn-node-metrics-cert\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073537 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qs7k\" (UniqueName: \"kubernetes.io/projected/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-kube-api-access-5qs7k\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073575 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7cc81f17-d52c-4edb-9f33-de2f37c80b54-crio-socket\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.073676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073639 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-config\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073707 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a1808a-9488-4d49-8028-4860660c0c9d-serving-cert\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073782 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-multus-certs\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073826 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073835 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a1808a-9488-4d49-8028-4860660c0c9d-serving-cert\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073845 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-snapshots\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073889 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-cnibin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073909 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-snapshots\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073914 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c95c406-8766-4a77-9639-513f2f37c1d3-cert\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.073948 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqj6s\" (UniqueName: \"kubernetes.io/projected/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff-kube-api-access-hqj6s\") pod \"kas-connection-checker-685895c7c9-xrnfv\" (UID: \"e0a3e17d-abf7-4d4b-afe0-fd028d2caeff\") " pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074009 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8t7\" (UniqueName: \"kubernetes.io/projected/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-kube-api-access-xr8t7\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.074042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074028 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-daemon-config\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074065 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jpx\" (UniqueName: \"kubernetes.io/projected/68e1c673-629b-4840-9f7a-aa64777cf32a-kube-api-access-j5jpx\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074097 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-kubelet\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074124 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pmx\" (UniqueName: \"kubernetes.io/projected/700b9d15-342a-46d1-b94e-6b44aae2d9b5-kube-api-access-g5pmx\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074151 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65f4d5dd-a512-425a-91b9-de7749f091c6-serving-cert\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074182 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwpg\" (UniqueName: \"kubernetes.io/projected/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-api-access-qqwpg\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074229 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-socket-dir-parent\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074262 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-serviceca\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:22:38.074322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074288 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65f4d5dd-a512-425a-91b9-de7749f091c6-serving-cert\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074325 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-sys\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074378 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-tmp\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074408 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074435 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5210f23-6d88-413f-b0b5-c1b938713ca9-tmp\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074436 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074465 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-host\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074495 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqnx\" (UniqueName: \"kubernetes.io/projected/35ea235a-c0ba-44b9-909f-e3dd472a6c36-kube-api-access-cgqnx\") pod \"kas-connection-checker-685895c7c9-hdlkm\" (UID: \"35ea235a-c0ba-44b9-909f-e3dd472a6c36\") " pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" Nov 03 22:22:38.074526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074524 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074533 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074547 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-ovn\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074583 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074599 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074603 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5210f23-6d88-413f-b0b5-c1b938713ca9-serving-cert\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074641 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-k8s-cni-cncf-io\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074666 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.074732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074707 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074717 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5210f23-6d88-413f-b0b5-c1b938713ca9-serving-cert\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074731 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/7c95c406-8766-4a77-9639-513f2f37c1d3-kube-api-access-sbfgm\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074766 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk4w\" (UniqueName: \"kubernetes.io/projected/60be4f8c-7051-444d-ad87-7e7895c94542-kube-api-access-hpk4w\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074815 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/458febd7-7dcd-4ef1-9b8b-27037557b753-iptables-alerter-script\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074833 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-etc-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074849 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-log-socket\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074868 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f621d2a-1926-4044-80bb-d64a0574e356-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:38.074966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.074939 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075047 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075095 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075134 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075179 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075202 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075218 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-node-log\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075234 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075251 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075293 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-registration-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075310 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-run\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.075337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075326 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e1c673-629b-4840-9f7a-aa64777cf32a-config-volume\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075354 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-netd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075372 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-env-overrides\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075387 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-hostroot\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075404 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8xp\" (UniqueName: \"kubernetes.io/projected/a9b5b425-585c-4fc9-bc11-2510c7035c70-kube-api-access-sx8xp\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075497 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qrs\" (UniqueName: \"kubernetes.io/projected/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34-kube-api-access-j9qrs\") pod \"network-check-source-6f78d59f99-7hx9j\" (UID: \"3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075636 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-var-lib-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075669 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-config\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075697 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075761 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-host\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:22:38.075827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075813 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075839 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075864 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-bin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075905 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-etc-kubernetes\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075926 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7234dd-7b73-427d-b0e1-afd778bd2028-tmp-dir\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075942 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.075988 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cnibin\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076020 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076060 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-cni-binary-copy\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076089 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-multus\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076114 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7234dd-7b73-427d-b0e1-afd778bd2028-tmp-dir\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076118 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-kubelet\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076159 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgpc9\" (UniqueName: \"kubernetes.io/projected/1f23b7eb-3f0e-4215-a42c-c20c976900dc-kube-api-access-cgpc9\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076191 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-netns\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076207 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-script-lib\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.076232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076224 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-conf\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076248 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076275 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-signing-key\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076303 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-service-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076347 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076380 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-sys-fs\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076408 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076458 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjrh\" (UniqueName: \"kubernetes.io/projected/458febd7-7dcd-4ef1-9b8b-27037557b753-kube-api-access-9wjrh\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076478 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076494 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-conf-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076510 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/458febd7-7dcd-4ef1-9b8b-27037557b753-host-slash\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076527 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-systemd-units\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076563 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076581 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f4d5dd-a512-425a-91b9-de7749f091c6-config\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076600 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-system-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076616 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-kubelet-config\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076637 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076678 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/add89f53-e47e-41de-8d0d-6fc06d826d29-kube-api-access-dstxj\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.076719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076700 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-var-lib-kubelet\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076813 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fh74\" (UniqueName: \"kubernetes.io/projected/b6fe9b11-1b54-4871-9236-dae9154b43fb-kube-api-access-4fh74\") pod \"volume-data-source-validator-58c9f5f46d-zwxcs\" (UID: \"b6fe9b11-1b54-4871-9236-dae9154b43fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076837 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-dbus\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076861 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076878 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-device-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076897 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cc81f17-d52c-4edb-9f33-de2f37c80b54-data-volume\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076931 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvnh\" (UniqueName: \"kubernetes.io/projected/3714e210-a948-4ba6-a174-0fc8eb8e06db-kube-api-access-bzvnh\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.076958 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-default-certificate\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077007 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cc81f17-d52c-4edb-9f33-de2f37c80b54-data-volume\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077092 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fkn\" (UniqueName: \"kubernetes.io/projected/e3a1808a-9488-4d49-8028-4860660c0c9d-kube-api-access-79fkn\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077119 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-kubernetes\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077136 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a374ff-7146-47f5-bcbd-6c4dc710d580-konnectivity-ca\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077157 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d7234dd-7b73-427d-b0e1-afd778bd2028-hosts-file\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077234 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x762b\" (UniqueName: \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077269 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077302 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92f9\" (UniqueName: \"kubernetes.io/projected/d5210f23-6d88-413f-b0b5-c1b938713ca9-kube-api-access-c92f9\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077330 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077364 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077479 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-stats-auth\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.077523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.077509 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.079128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.079113 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5wr4r\"" Nov 03 22:22:38.079296 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.079280 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:22:38.079353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.079312 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:22:38.080696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.080683 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:22:38.080927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.080892 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:22:38.081013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.081001 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:22:38.082464 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.082449 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:22:38.083021 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.082998 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:22:38.083073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083025 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:22:38.083177 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083107 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:22:38.083272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083226 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:22:38.083331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083279 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:22:38.083331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083291 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bwztx\"" Nov 03 22:22:38.083331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083283 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hp7n8\"" Nov 03 22:22:38.083331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083306 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:22:38.083501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083348 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nh9f8\"" Nov 03 22:22:38.083501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083396 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ljhlr\"" Nov 03 22:22:38.083501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083415 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6sdx\"" Nov 03 22:22:38.083703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083519 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ztkgs\"" Nov 03 22:22:38.083703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083590 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mdpv5\"" Nov 03 22:22:38.083703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083604 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-khx6v\"" Nov 03 22:22:38.083703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083626 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v9lfz\"" Nov 03 22:22:38.083703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083677 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zvxhn\"" Nov 03 22:22:38.083927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083769 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hvj2k\"" Nov 03 22:22:38.083927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083803 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kccnl\"" Nov 03 22:22:38.083927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083835 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724a681a-c911-439e-aa2c-904081578713-ovn-node-metrics-cert\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.083927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083853 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kfchj\"" Nov 03 22:22:38.083927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083902 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk9bw\"" Nov 03 22:22:38.084065 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.083963 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:22:38.084065 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.084005 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2dn77\"" Nov 03 22:22:38.084116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.084084 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kbfvs\"" Nov 03 22:22:38.084158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.084137 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:22:38.084188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.084162 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c95c406-8766-4a77-9639-513f2f37c1d3-cert\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:38.084218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.084194 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2jqrs\"" Nov 03 22:22:38.085844 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.085829 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.086938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.086925 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.087996 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.087984 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.088069 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.088050 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-stats-auth\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.088100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.088068 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-default-certificate\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.093246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.093229 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.093399 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.093383 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06b3993b-eaa8-4649-9c38-2b9bb7a22498-original-pull-secret\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.093432 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.093399 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a374ff-7146-47f5-bcbd-6c4dc710d580-agent-certs\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:38.106505 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.106485 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:22:38.106588 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.106577 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 22:22:38.106736 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.106712 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vkjzf\"" Nov 03 22:22:38.106834 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.106740 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-klff9\"" Nov 03 22:22:38.106834 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.106811 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jvtm5\"" Nov 03 22:22:38.140884 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.140858 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.148695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.148675 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwt5\" (UniqueName: \"kubernetes.io/projected/e3939775-883b-454d-9e3b-6949f6984dfb-kube-api-access-5gwt5\") pod \"kas-connection-checker-685895c7c9-6ddbh\" (UID: \"e3939775-883b-454d-9e3b-6949f6984dfb\") " pod="kube-system/kas-connection-checker-685895c7c9-6ddbh" Nov 03 22:22:38.152037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.152019 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.157989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.157970 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqnx\" (UniqueName: \"kubernetes.io/projected/35ea235a-c0ba-44b9-909f-e3dd472a6c36-kube-api-access-cgqnx\") pod \"kas-connection-checker-685895c7c9-hdlkm\" (UID: \"35ea235a-c0ba-44b9-909f-e3dd472a6c36\") " pod="kube-system/kas-connection-checker-685895c7c9-hdlkm" Nov 03 22:22:38.158059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.157998 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqj6s\" (UniqueName: \"kubernetes.io/projected/e0a3e17d-abf7-4d4b-afe0-fd028d2caeff-kube-api-access-hqj6s\") pod \"kas-connection-checker-685895c7c9-xrnfv\" (UID: \"e0a3e17d-abf7-4d4b-afe0-fd028d2caeff\") " pod="kube-system/kas-connection-checker-685895c7c9-xrnfv" Nov 03 22:22:38.161588 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161565 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.161733 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161719 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.161900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161836 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:22:38.161900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161834 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.161900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161859 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.161900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161874 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.161900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161881 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161912 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161982 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.161998 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162010 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162030 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162032 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162032 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162117 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:22:38.162412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162193 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162194 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162208 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.162412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162227 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162256 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162538 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:22:38.162563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162551 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162566 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.162636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162576 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:22:38.162636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162583 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162653 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:22:38.162747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162670 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:22:38.162747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162681 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:22:38.162747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162702 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162716 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162951 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162755 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.162951 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162882 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 22:22:38.162951 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.162913 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:22:38.163069 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163054 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.163142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163071 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:22:38.163142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163100 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163116 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163229 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:38.163265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163259 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163332 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163284 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.163591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163571 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163697 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163599 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.163741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163697 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163702 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvnh\" (UniqueName: \"kubernetes.io/projected/3714e210-a948-4ba6-a174-0fc8eb8e06db-kube-api-access-bzvnh\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.163741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163724 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.163904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163759 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/7c95c406-8766-4a77-9639-513f2f37c1d3-kube-api-access-sbfgm\") pod \"ingress-canary-pqv9b\" (UID: \"7c95c406-8766-4a77-9639-513f2f37c1d3\") " pod="openshift-ingress-canary/ingress-canary-pqv9b" Nov 03 22:22:38.163904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163833 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163832 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163866 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:22:38.163904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163882 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-config\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.163904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163894 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.163904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163873 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163929 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163938 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.163972 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164029 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164054 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164068 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164160 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.164250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164197 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:22:38.164563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164394 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-daemon-config\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.164563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164547 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-serviceca\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:22:38.164698 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.164680 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghp7z\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.165181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.165162 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f621d2a-1926-4044-80bb-d64a0574e356-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:38.165259 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.165208 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/458febd7-7dcd-4ef1-9b8b-27037557b753-iptables-alerter-script\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:22:38.166321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166306 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e1c673-629b-4840-9f7a-aa64777cf32a-config-volume\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.166377 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166329 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-config\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.166377 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166335 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.166488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166478 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9b5b425-585c-4fc9-bc11-2510c7035c70-cni-binary-copy\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.166540 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166488 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-env-overrides\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.166540 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166477 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.166606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166536 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.166606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166539 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724a681a-c911-439e-aa2c-904081578713-ovnkube-script-lib\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.166606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166564 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-service-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.166799 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166767 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jpx\" (UniqueName: \"kubernetes.io/projected/68e1c673-629b-4840-9f7a-aa64777cf32a-kube-api-access-j5jpx\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.166923 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166907 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xjb\" (UniqueName: \"kubernetes.io/projected/64ebd614-43bd-479e-8e95-c33b2c93361c-kube-api-access-g6xjb\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.166961 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166937 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.166990 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.166960 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l29j\" (UniqueName: \"kubernetes.io/projected/65f4d5dd-a512-425a-91b9-de7749f091c6-kube-api-access-4l29j\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168202 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnfb\" (UniqueName: \"kubernetes.io/projected/4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34-kube-api-access-kpnfb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6\" (UID: \"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168238 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qs7k\" (UniqueName: \"kubernetes.io/projected/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-kube-api-access-5qs7k\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168269 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgpc9\" (UniqueName: \"kubernetes.io/projected/1f23b7eb-3f0e-4215-a42c-c20c976900dc-kube-api-access-cgpc9\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168339 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxd8q\" (UniqueName: \"kubernetes.io/projected/1d7234dd-7b73-427d-b0e1-afd778bd2028-kube-api-access-fxd8q\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168402 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x762b\" (UniqueName: \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168525 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fkn\" (UniqueName: \"kubernetes.io/projected/e3a1808a-9488-4d49-8028-4860660c0c9d-kube-api-access-79fkn\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168529 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92f9\" (UniqueName: \"kubernetes.io/projected/d5210f23-6d88-413f-b0b5-c1b938713ca9-kube-api-access-c92f9\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.168555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.168529 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.170852 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.170834 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqkb\" (UniqueName: \"kubernetes.io/projected/724a681a-c911-439e-aa2c-904081578713-kube-api-access-dsqkb\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.170924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.170857 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vq5n\" (UniqueName: \"kubernetes.io/projected/de375da1-1b02-4796-ad49-2062bc5bc74b-kube-api-access-8vq5n\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:38.170924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.170865 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwpg\" (UniqueName: \"kubernetes.io/projected/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-api-access-qqwpg\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.171058 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.171037 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8xp\" (UniqueName: \"kubernetes.io/projected/a9b5b425-585c-4fc9-bc11-2510c7035c70-kube-api-access-sx8xp\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.171115 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.171100 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fh74\" (UniqueName: \"kubernetes.io/projected/b6fe9b11-1b54-4871-9236-dae9154b43fb-kube-api-access-4fh74\") pod \"volume-data-source-validator-58c9f5f46d-zwxcs\" (UID: \"b6fe9b11-1b54-4871-9236-dae9154b43fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zwxcs" Nov 03 22:22:38.172192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.172176 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pmx\" (UniqueName: \"kubernetes.io/projected/700b9d15-342a-46d1-b94e-6b44aae2d9b5-kube-api-access-g5pmx\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.172664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.172649 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dskzm\" (UniqueName: \"kubernetes.io/projected/e8d3725a-cbd4-4ef3-a433-2010c83a5f3d-kube-api-access-dskzm\") pod \"network-check-target-mzwbr\" (UID: \"e8d3725a-cbd4-4ef3-a433-2010c83a5f3d\") " pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:38.174064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.174032 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.174146 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.174094 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.174325 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.174305 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjrh\" (UniqueName: \"kubernetes.io/projected/458febd7-7dcd-4ef1-9b8b-27037557b753-kube-api-access-9wjrh\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:22:38.174382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.174358 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/add89f53-e47e-41de-8d0d-6fc06d826d29-kube-api-access-dstxj\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.174474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.174458 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qrs\" (UniqueName: \"kubernetes.io/projected/3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34-kube-api-access-j9qrs\") pod \"network-check-source-6f78d59f99-7hx9j\" (UID: \"3bc089dd-eb07-47b1-a9bf-e1bc4e44aa34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hx9j" Nov 03 22:22:38.174576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.174559 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk4w\" (UniqueName: \"kubernetes.io/projected/60be4f8c-7051-444d-ad87-7e7895c94542-kube-api-access-hpk4w\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178007 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7cc81f17-d52c-4edb-9f33-de2f37c80b54-crio-socket\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178102 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7cc81f17-d52c-4edb-9f33-de2f37c80b54-crio-socket\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178103 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-multus-certs\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178151 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178164 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-multus-certs\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178182 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-cnibin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178214 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8t7\" (UniqueName: \"kubernetes.io/projected/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-kube-api-access-xr8t7\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178239 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-kubelet\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178272 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-socket-dir-parent\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.178280 4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178301 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-sys\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178332 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.178364 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.678344879 +0000 UTC m=+1.891113927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : secret "metrics-daemon-secret" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178391 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-host\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.178426 4898 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178460 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-ovn\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.178492 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.678475695 +0000 UTC m=+1.891244749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178543 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-cnibin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178604 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-socket-dir-parent\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178649 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-kubelet\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178422 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-ovn\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178692 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-k8s-cni-cncf-io\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178723 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178755 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-etc-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178761 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-host\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178778 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-log-socket\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178832 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178844 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-sys\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178841 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-k8s-cni-cncf-io\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178861 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178881 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-etc-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178895 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178896 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-log-socket\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.178964 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.178990 4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179004 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179037 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179049 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179095 4898 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179119 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.679103259 +0000 UTC m=+1.891872305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179137 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-node-log\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.179140 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179172 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.679157706 +0000 UTC m=+1.891926769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179221 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-node-log\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179264 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.679253871 +0000 UTC m=+1.892022915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179281 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-registration-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179293 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.679282732 +0000 UTC m=+1.892051797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : configmap "kube-rbac-proxy" not found Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179323 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-run\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179340 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-registration-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179352 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-netd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179385 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-hostroot\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179407 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-run\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179410 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-var-lib-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179448 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-var-lib-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179448 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179489 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-host\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179519 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179522 4898 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179549 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-bin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.179571 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.679552385 +0000 UTC m=+1.892321414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179588 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-bin\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179598 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-etc-kubernetes\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179629 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-netd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179631 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cnibin\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179666 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-multus\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179673 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-cnibin\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179698 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-kubelet\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179716 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-etc-kubernetes\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179729 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-netns\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179759 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-conf\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179784 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-signing-key\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179863 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179887 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179894 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-sys-fs\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179925 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179940 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-hostroot\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179950 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-conf-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179981 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/458febd7-7dcd-4ef1-9b8b-27037557b753-host-slash\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.179987 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd4817e-ed9f-4fea-858a-be4d25adcc1f-host\") pod \"node-ca-2g542\" (UID: \"dbd4817e-ed9f-4fea-858a-be4d25adcc1f\") " pod="openshift-image-registry/node-ca-2g542" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180011 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-systemd-units\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180041 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-system-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180071 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-kubelet-config\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180096 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-var-lib-kubelet\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180102 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysctl-conf\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180124 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-dbus\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180146 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-cni-multus\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180156 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-device-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180231 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-system-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180245 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-kubelet-config\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180272 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-sys-fs\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180296 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-var-lib-kubelet\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180311 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-var-lib-kubelet\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180352 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-netns\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180360 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180423 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06b3993b-eaa8-4649-9c38-2b9bb7a22498-dbus\") pod \"global-pull-secret-syncer-c9btg\" (UID: \"06b3993b-eaa8-4649-9c38-2b9bb7a22498\") " pod="kube-system/global-pull-secret-syncer-c9btg" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180443 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-device-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180472 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/458febd7-7dcd-4ef1-9b8b-27037557b753-host-slash\") pod \"iptables-alerter-qsnsn\" (UID: \"458febd7-7dcd-4ef1-9b8b-27037557b753\") " pod="openshift-network-operator/iptables-alerter-qsnsn" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180478 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-kubernetes\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180510 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d7234dd-7b73-427d-b0e1-afd778bd2028-hosts-file\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180514 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-systemd-units\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180555 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180563 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-kubernetes\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180589 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180602 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-openvswitch\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180670 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180707 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180769 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-cni-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180843 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d7234dd-7b73-427d-b0e1-afd778bd2028-hosts-file\") pod \"node-resolver-fqr6l\" (UID: \"1d7234dd-7b73-427d-b0e1-afd778bd2028\") " pod="openshift-dns/node-resolver-fqr6l" Nov 03 22:22:38.180915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.180884 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-multus-conf-dir\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.181191 4898 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.182160 4898 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182176 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-system-cni-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182217 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-netns\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.182226 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.682212316 +0000 UTC m=+1.894981347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.182233 4898 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182264 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182272 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5b425-585c-4fc9-bc11-2510c7035c70-host-run-netns\") pod \"multus-pjjqw\" (UID: \"a9b5b425-585c-4fc9-bc11-2510c7035c70\") " pod="openshift-multus/multus-pjjqw" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182279 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-system-cni-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.182290 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.682276841 +0000 UTC m=+1.895045888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182320 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182382 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182408 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f23b7eb-3f0e-4215-a42c-c20c976900dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqclr\" (UID: \"1f23b7eb-3f0e-4215-a42c-c20c976900dc\") " pod="openshift-multus/multus-additional-cni-plugins-jqclr" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182426 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-slash\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182454 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-systemd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182475 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182580 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-run-systemd\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182615 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182620 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182655 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-bin\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.182673 4898 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.182721 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.682707725 +0000 UTC m=+1.895476754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182736 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xd6w\" (UniqueName: \"kubernetes.io/projected/28365524-7763-4515-93fd-379ea89ffd5e-kube-api-access-8xd6w\") pod \"migrator-66df5d496b-72zn6\" (UID: \"28365524-7763-4515-93fd-379ea89ffd5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182759 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-cni-bin\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182773 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-lib-modules\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182837 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-modprobe-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182872 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182901 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-socket-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182906 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-lib-modules\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.182951 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-modprobe-d\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183003 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724a681a-c911-439e-aa2c-904081578713-host-slash\") pod \"ovnkube-node-rhx44\" (UID: \"724a681a-c911-439e-aa2c-904081578713\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.183007 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183119 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysconfig\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183013 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-sysconfig\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.183445 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183087 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/add89f53-e47e-41de-8d0d-6fc06d826d29-socket-dir\") pod \"aws-ebs-csi-driver-node-d4k92\" (UID: \"add89f53-e47e-41de-8d0d-6fc06d826d29\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4k92" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.183549 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:38.683532363 +0000 UTC m=+1.896301392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183546 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183619 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-systemd\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183667 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pw8n\" (UniqueName: \"kubernetes.io/projected/2ec50892-7849-4b30-934a-c9983238c501-kube-api-access-6pw8n\") pod \"migrator-66df5d496b-m2dcb\" (UID: \"2ec50892-7849-4b30-934a-c9983238c501\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" Nov 03 22:22:38.183707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183682 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3714e210-a948-4ba6-a174-0fc8eb8e06db-etc-systemd\") pod \"tuned-zg9h5\" (UID: \"3714e210-a948-4ba6-a174-0fc8eb8e06db\") " pod="openshift-cluster-node-tuning-operator/tuned-zg9h5" Nov 03 22:22:38.185052 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.183919 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-signing-key\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.195825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.195783 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:22:38.195904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.195883 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:22:38.197136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.197115 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1808a-9488-4d49-8028-4860660c0c9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-thjpw\" (UID: \"e3a1808a-9488-4d49-8028-4860660c0c9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:38.197202 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.197122 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5210f23-6d88-413f-b0b5-c1b938713ca9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z5lhd\" (UID: \"d5210f23-6d88-413f-b0b5-c1b938713ca9\") " pod="openshift-insights/insights-operator-7c4986d449-z5lhd" Nov 03 22:22:38.198191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.198174 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:22:38.198269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.198216 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:22:38.200778 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.200763 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:22:38.203488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.203472 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60be4f8c-7051-444d-ad87-7e7895c94542-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.205860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.205846 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\") pod \"periodic-gathering-fnl7t-pdbr9\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:22:38.207273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.207260 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f4d5dd-a512-425a-91b9-de7749f091c6-config\") pod \"service-ca-operator-544d749c8c-7khh5\" (UID: \"65f4d5dd-a512-425a-91b9-de7749f091c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" Nov 03 22:22:38.207379 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.207368 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a374ff-7146-47f5-bcbd-6c4dc710d580-konnectivity-ca\") pod \"konnectivity-agent-pvpqf\" (UID: \"c1a374ff-7146-47f5-bcbd-6c4dc710d580\") " pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:38.212261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.212245 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:22:38.214519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.214502 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.215667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.215651 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.239648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.239598 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8t7\" (UniqueName: \"kubernetes.io/projected/9b6ff0e6-235b-4650-b5a8-54117cb73b3c-kube-api-access-xr8t7\") pod \"service-ca-5fffc97bdf-m5pmk\" (UID: \"9b6ff0e6-235b-4650-b5a8-54117cb73b3c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.245100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.245081 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pw8n\" (UniqueName: \"kubernetes.io/projected/2ec50892-7849-4b30-934a-c9983238c501-kube-api-access-6pw8n\") pod \"migrator-66df5d496b-m2dcb\" (UID: \"2ec50892-7849-4b30-934a-c9983238c501\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" Nov 03 22:22:38.245183 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.245172 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xd6w\" (UniqueName: \"kubernetes.io/projected/28365524-7763-4515-93fd-379ea89ffd5e-kube-api-access-8xd6w\") pod \"migrator-66df5d496b-72zn6\" (UID: \"28365524-7763-4515-93fd-379ea89ffd5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" Nov 03 22:22:38.310423 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.310393 4898 scope.go:122] "RemoveContainer" containerID="fbf7747233d82ae2ff25e3a729e1ddf99197402b0e6cff13442a2475f1cc27bf" Nov 03 22:22:38.335895 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.335875 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" Nov 03 22:22:38.342174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.342151 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" Nov 03 22:22:38.347575 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.347552 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" Nov 03 22:22:38.688128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688096 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:38.688128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688133 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688169 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688207 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688241 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688254 4898 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688262 4898 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688279 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688313 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688296591 +0000 UTC m=+2.901065637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688317 4898 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688341 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688321361 +0000 UTC m=+2.901090403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688347 4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688368 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688385 4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688396 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688384664 +0000 UTC m=+2.901153699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : secret "metrics-daemon-secret" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688345 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688414 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688420 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688414153 +0000 UTC m=+2.901183182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688351 4898 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:38.688442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688435 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688426889 +0000 UTC m=+2.901195918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688461 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688453447 +0000 UTC m=+2.901222475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688488 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688482771 +0000 UTC m=+2.901251801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688501 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688521 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688537 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688579 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688586 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688609 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688600868 +0000 UTC m=+2.901369902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.688584 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688631 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688617172 +0000 UTC m=+2.901386218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688640 4898 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688636 4898 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688667 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.688656907 +0000 UTC m=+2.901425958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : configmap "kube-rbac-proxy" not found Nov 03 22:22:38.689098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.688692 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:39.68868292 +0000 UTC m=+2.901451965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:38.713869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.713852 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6"] Nov 03 22:22:38.716257 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:38.716229 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28365524_7763_4515_93fd_379ea89ffd5e.slice/crio-57919e80f3f84211dbb2ca78ab52553e3842c6026c0402905717327745f6103f WatchSource:0}: Error finding container 57919e80f3f84211dbb2ca78ab52553e3842c6026c0402905717327745f6103f: Status 404 returned error can't find the container with id 57919e80f3f84211dbb2ca78ab52553e3842c6026c0402905717327745f6103f Nov 03 22:22:38.718297 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.718283 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:22:38.761457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.761431 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb"] Nov 03 22:22:38.764571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.764509 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m5pmk"] Nov 03 22:22:38.765195 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:38.765172 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec50892_7849_4b30_934a_c9983238c501.slice/crio-0307c4b03c2ba01a1d0e10e31b0fccb70992e3888132a6e4010b6349d227d417 WatchSource:0}: Error finding container 0307c4b03c2ba01a1d0e10e31b0fccb70992e3888132a6e4010b6349d227d417: Status 404 returned error can't find the container with id 0307c4b03c2ba01a1d0e10e31b0fccb70992e3888132a6e4010b6349d227d417 Nov 03 22:22:38.769252 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:38.769217 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6ff0e6_235b_4650_b5a8_54117cb73b3c.slice/crio-8813804592c95193ab5d8db4aa31c4153eb8e962549735201253852dfdf2ad5d WatchSource:0}: Error finding container 8813804592c95193ab5d8db4aa31c4153eb8e962549735201253852dfdf2ad5d: Status 404 returned error can't find the container with id 8813804592c95193ab5d8db4aa31c4153eb8e962549735201253852dfdf2ad5d Nov 03 22:22:38.976192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.976112 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" event={"ID":"2ec50892-7849-4b30-934a-c9983238c501","Type":"ContainerStarted","Data":"0307c4b03c2ba01a1d0e10e31b0fccb70992e3888132a6e4010b6349d227d417"} Nov 03 22:22:38.977288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.977258 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" event={"ID":"28365524-7763-4515-93fd-379ea89ffd5e","Type":"ContainerStarted","Data":"57919e80f3f84211dbb2ca78ab52553e3842c6026c0402905717327745f6103f"} Nov 03 22:22:38.978670 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.978644 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" event={"ID":"9b6ff0e6-235b-4650-b5a8-54117cb73b3c","Type":"ContainerStarted","Data":"7b9c65449d288ae4b44a68dfec4c91d62f10fed765d61edb410e6a1cdede6366"} Nov 03 22:22:38.978758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.978683 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" event={"ID":"9b6ff0e6-235b-4650-b5a8-54117cb73b3c","Type":"ContainerStarted","Data":"8813804592c95193ab5d8db4aa31c4153eb8e962549735201253852dfdf2ad5d"} Nov 03 22:22:38.980161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.980145 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:22:38.980499 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.980487 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/1.log" Nov 03 22:22:38.980559 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.980516 4898 generic.go:356] "Generic (PLEG): container finished" podID="e3a1808a-9488-4d49-8028-4860660c0c9d" containerID="1db13483c24c13a859ac1ff207be20a6f060c42aa749aed73c33867b1c800169" exitCode=255 Nov 03 22:22:38.980612 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.980594 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" event={"ID":"e3a1808a-9488-4d49-8028-4860660c0c9d","Type":"ContainerDied","Data":"1db13483c24c13a859ac1ff207be20a6f060c42aa749aed73c33867b1c800169"} Nov 03 22:22:38.980654 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.980642 4898 scope.go:122] "RemoveContainer" containerID="fbf7747233d82ae2ff25e3a729e1ddf99197402b0e6cff13442a2475f1cc27bf" Nov 03 22:22:38.981243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:38.981222 4898 scope.go:122] "RemoveContainer" containerID="1db13483c24c13a859ac1ff207be20a6f060c42aa749aed73c33867b1c800169" Nov 03 22:22:38.981403 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:38.981385 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-thjpw_openshift-console-operator(e3a1808a-9488-4d49-8028-4860660c0c9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" podUID="e3a1808a-9488-4d49-8028-4860660c0c9d" Nov 03 22:22:39.700753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.700725 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:39.700952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.700761 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:39.700952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.700811 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:39.700952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.700840 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:39.700952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.700865 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:39.700952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.700900 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:39.700952 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.700940 4898 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.700948 4898 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.700978 4898 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.700940 4898 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.700958 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.700981 4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701018 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.700997 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls podName:7cc81f17-d52c-4edb-9f33-de2f37c80b54 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.700983887 +0000 UTC m=+4.913752919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8md2" (UID: "7cc81f17-d52c-4edb-9f33-de2f37c80b54") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.701066 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.701107 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701121 4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701137 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701121819 +0000 UTC m=+4.913890851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701160 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701147144 +0000 UTC m=+4.913916186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701178 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:39.701186 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701189 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701178 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls podName:60be4f8c-7051-444d-ad87-7e7895c94542 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701168392 +0000 UTC m=+4.913937427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s7gxg" (UID: "60be4f8c-7051-444d-ad87-7e7895c94542") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701222 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701212166 +0000 UTC m=+4.913981204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701241 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701232537 +0000 UTC m=+4.914001567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : secret "metrics-daemon-secret" not found Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701257 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701250323 +0000 UTC m=+4.914019390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.701280 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701305 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701293624 +0000 UTC m=+4.914062664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701337 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701327632 +0000 UTC m=+4.914096681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.701334 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.701406 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701510 4898 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:39.701629 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:39.701543 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert podName:7f621d2a-1926-4044-80bb-d64a0574e356 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:41.701533348 +0000 UTC m=+4.914302395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jkxgz" (UID: "7f621d2a-1926-4044-80bb-d64a0574e356") : secret "networking-console-plugin-cert" not found Nov 03 22:22:39.702037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.701728 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7cc81f17-d52c-4edb-9f33-de2f37c80b54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:39.989045 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:39.988975 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:22:40.320026 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:40.319989 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fqr6l_1d7234dd-7b73-427d-b0e1-afd778bd2028/dns-node-resolver/0.log" Nov 03 22:22:40.664339 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:40.664272 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-m5pmk" podStartSLOduration=3.664258795 podStartE2EDuration="3.664258795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:40.663625299 +0000 UTC m=+3.876394354" watchObservedRunningTime="2026-11-03 22:22:40.664258795 +0000 UTC m=+3.877027846" Nov 03 22:22:40.994670 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:40.994578 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" event={"ID":"28365524-7763-4515-93fd-379ea89ffd5e","Type":"ContainerStarted","Data":"1ae74d90bb65370fcbef13522827cd2afcd4ffabd6cbfdb56d127f9998cee5d2"} Nov 03 22:22:40.996292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:40.996261 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" event={"ID":"2ec50892-7849-4b30-934a-c9983238c501","Type":"ContainerStarted","Data":"8df630b6fcea86528310c729845225a2c4281375e5b15b7cd3c600fd2407f624"} Nov 03 22:22:41.110248 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.110196 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-72zn6" podStartSLOduration=3.325999569 podStartE2EDuration="5.110180792s" totalImagesPullingTime="1.784181223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:41.109576048 +0000 UTC m=+4.322345098" watchObservedRunningTime="2026-11-03 22:22:41.110180792 +0000 UTC m=+4.322949839" Nov 03 22:22:41.120062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.120038 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2g542_dbd4817e-ed9f-4fea-858a-be4d25adcc1f/node-ca/0.log" Nov 03 22:22:41.246883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.246775 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m2dcb" podStartSLOduration=3.481597097 podStartE2EDuration="5.246762011s" totalImagesPullingTime="1.765164914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:41.24623065 +0000 UTC m=+4.458999714" watchObservedRunningTime="2026-11-03 22:22:41.246762011 +0000 UTC m=+4.459531055" Nov 03 22:22:41.719753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.719715 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.719761 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.719830 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.719866 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.719877 4898 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.719898 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.719933 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.719943 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.719928727 +0000 UTC m=+8.932697757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:41.719977 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.719879 4898 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720001 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.719991031 +0000 UTC m=+8.932760063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : secret "router-metrics-certs-default" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720024 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720035 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720043 4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720066 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.720057688 +0000 UTC m=+8.932826747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.720072 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720099 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs podName:de375da1-1b02-4796-ad49-2062bc5bc74b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.720084601 +0000 UTC m=+8.932853635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs") pod "network-metrics-daemon-rcrd8" (UID: "de375da1-1b02-4796-ad49-2062bc5bc74b") : secret "metrics-daemon-secret" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720158 4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720206 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.720193701 +0000 UTC m=+8.932962744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.720234 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.720276 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.720328 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720367 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720384 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:41.720473 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720436 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.720422914 +0000 UTC m=+8.933191959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:41.721311 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:41.720591 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.720537155 +0000 UTC m=+8.933306202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:41.722402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.722369 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7cc81f17-d52c-4edb-9f33-de2f37c80b54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8md2\" (UID: \"7cc81f17-d52c-4edb-9f33-de2f37c80b54\") " pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:41.722539 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.722508 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60be4f8c-7051-444d-ad87-7e7895c94542-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s7gxg\" (UID: \"60be4f8c-7051-444d-ad87-7e7895c94542\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:41.722649 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.722628 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f621d2a-1926-4044-80bb-d64a0574e356-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jkxgz\" (UID: \"7f621d2a-1926-4044-80bb-d64a0574e356\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:41.912213 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.912178 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" Nov 03 22:22:41.913757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.913730 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" Nov 03 22:22:41.919741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.919698 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqv9b_7c95c406-8766-4a77-9639-513f2f37c1d3/serve-healthcheck-canary/0.log" Nov 03 22:22:41.921804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:41.921771 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r8md2" Nov 03 22:22:42.082656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.082623 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:22:42.121906 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.121881 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-72zn6_28365524-7763-4515-93fd-379ea89ffd5e/migrator/0.log" Nov 03 22:22:42.169410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.169382 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:42.169769 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.169756 4898 scope.go:122] "RemoveContainer" containerID="1db13483c24c13a859ac1ff207be20a6f060c42aa749aed73c33867b1c800169" Nov 03 22:22:42.169977 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:42.169957 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-thjpw_openshift-console-operator(e3a1808a-9488-4d49-8028-4860660c0c9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" podUID="e3a1808a-9488-4d49-8028-4860660c0c9d" Nov 03 22:22:42.320766 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.320736 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-m2dcb_2ec50892-7849-4b30-934a-c9983238c501/migrator/0.log" Nov 03 22:22:42.362140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.362106 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jkxgz"] Nov 03 22:22:42.365194 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.365167 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg"] Nov 03 22:22:42.365310 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:42.365222 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f621d2a_1926_4044_80bb_d64a0574e356.slice/crio-ba231e22b50a841b0828639d8f7f1ff5c1469506c21ee75ae64899bf956fd80d WatchSource:0}: Error finding container ba231e22b50a841b0828639d8f7f1ff5c1469506c21ee75ae64899bf956fd80d: Status 404 returned error can't find the container with id ba231e22b50a841b0828639d8f7f1ff5c1469506c21ee75ae64899bf956fd80d Nov 03 22:22:42.366129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.366101 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r8md2"] Nov 03 22:22:42.404660 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:42.404639 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc81f17_d52c_4edb_9f33_de2f37c80b54.slice/crio-a464b5df1a51bd73d7e29e0fc558583d5865cec0ae73ff214ecddb05ad466441 WatchSource:0}: Error finding container a464b5df1a51bd73d7e29e0fc558583d5865cec0ae73ff214ecddb05ad466441: Status 404 returned error can't find the container with id a464b5df1a51bd73d7e29e0fc558583d5865cec0ae73ff214ecddb05ad466441 Nov 03 22:22:42.412665 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:42.412642 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60be4f8c_7051_444d_ad87_7e7895c94542.slice/crio-745b7c142d9264a6eafdea48fde093e6a6a03351533ef4ae636bdf80ea83d7e8 WatchSource:0}: Error finding container 745b7c142d9264a6eafdea48fde093e6a6a03351533ef4ae636bdf80ea83d7e8: Status 404 returned error can't find the container with id 745b7c142d9264a6eafdea48fde093e6a6a03351533ef4ae636bdf80ea83d7e8 Nov 03 22:22:42.520570 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.520536 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6_4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34/kube-storage-version-migrator-operator/0.log" Nov 03 22:22:42.931446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:42.931417 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:43.006007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:43.005970 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" event={"ID":"60be4f8c-7051-444d-ad87-7e7895c94542","Type":"ContainerStarted","Data":"745b7c142d9264a6eafdea48fde093e6a6a03351533ef4ae636bdf80ea83d7e8"} Nov 03 22:22:43.007436 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:43.007410 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8md2" event={"ID":"7cc81f17-d52c-4edb-9f33-de2f37c80b54","Type":"ContainerStarted","Data":"9a678b9c9943d2219628159f73d08698474a14eb8a4930e0f1763ddbbb387e34"} Nov 03 22:22:43.007566 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:43.007465 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8md2" event={"ID":"7cc81f17-d52c-4edb-9f33-de2f37c80b54","Type":"ContainerStarted","Data":"a464b5df1a51bd73d7e29e0fc558583d5865cec0ae73ff214ecddb05ad466441"} Nov 03 22:22:43.008562 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:43.008539 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" event={"ID":"7f621d2a-1926-4044-80bb-d64a0574e356","Type":"ContainerStarted","Data":"ba231e22b50a841b0828639d8f7f1ff5c1469506c21ee75ae64899bf956fd80d"} Nov 03 22:22:44.015201 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:44.015161 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8md2" event={"ID":"7cc81f17-d52c-4edb-9f33-de2f37c80b54","Type":"ContainerStarted","Data":"f1711e63df02599cc91964232174b31d620925d0fca77e1bbd94b63266314689"} Nov 03 22:22:44.054098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:44.054032 4898 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:44.054491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:44.054476 4898 scope.go:122] "RemoveContainer" containerID="1db13483c24c13a859ac1ff207be20a6f060c42aa749aed73c33867b1c800169" Nov 03 22:22:44.054723 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:44.054695 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-thjpw_openshift-console-operator(e3a1808a-9488-4d49-8028-4860660c0c9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" podUID="e3a1808a-9488-4d49-8028-4860660c0c9d" Nov 03 22:22:45.021942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.021850 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s7gxg" event={"ID":"60be4f8c-7051-444d-ad87-7e7895c94542","Type":"ContainerStarted","Data":"f36c9e8b7bb5ecfae280458bc4e984cc7ec0ebe167f5512296b8e9c9a80c1d91"} Nov 03 22:22:45.023561 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.023530 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jkxgz" event={"ID":"7f621d2a-1926-4044-80bb-d64a0574e356","Type":"ContainerStarted","Data":"546d60c29d778c8c7c81087e9e71e295d9e7422ea023098cd06feb956753c20a"} Nov 03 22:22:45.761393 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.761350 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:45.761587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.761427 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:45.761587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.761461 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:45.761587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.761501 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:45.761587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.761530 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:45.761587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.761554 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:45.761587 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761556 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:45.761587 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761581 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d49cb494-tnd66: secret "image-registry-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.761627 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761650 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls podName:07e5df88-99b0-4026-8ae2-b71752948379 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:53.761629333 +0000 UTC m=+16.974398381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls") pod "image-registry-58d49cb494-tnd66" (UID: "07e5df88-99b0-4026-8ae2-b71752948379") : secret "image-registry-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761679 4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761704 4898 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761682 4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761754 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls podName:68e1c673-629b-4840-9f7a-aa64777cf32a nodeName:}" failed. No retries permitted until 2026-11-03 22:22:53.761736509 +0000 UTC m=+16.974505538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls") pod "dns-default-jmjbp" (UID: "68e1c673-629b-4840-9f7a-aa64777cf32a") : secret "dns-default-metrics-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761760 4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d9b64b6dc-xqdzm: secret "image-registry-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761774 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls podName:64ebd614-43bd-479e-8e95-c33b2c93361c nodeName:}" failed. No retries permitted until 2026-11-03 22:22:53.761763782 +0000 UTC m=+16.974532818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kb8lw" (UID: "64ebd614-43bd-479e-8e95-c33b2c93361c") : secret "samples-operator-tls" not found Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761804 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle podName:700b9d15-342a-46d1-b94e-6b44aae2d9b5 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:53.761780572 +0000 UTC m=+16.974549607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle") pod "router-default-74d647c79-nsnqv" (UID: "700b9d15-342a-46d1-b94e-6b44aae2d9b5") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:45.761931 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:45.761820 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls podName:3b569914-42dd-4a2f-bd2c-63b37b3c47c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:53.761812206 +0000 UTC m=+16.974581237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls") pod "image-registry-5d9b64b6dc-xqdzm" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9") : secret "image-registry-tls" not found Nov 03 22:22:45.764151 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.764096 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de375da1-1b02-4796-ad49-2062bc5bc74b-metrics-certs\") pod \"network-metrics-daemon-rcrd8\" (UID: \"de375da1-1b02-4796-ad49-2062bc5bc74b\") " pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:45.764151 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.764141 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700b9d15-342a-46d1-b94e-6b44aae2d9b5-metrics-certs\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:45.816983 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:45.816951 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcrd8" Nov 03 22:22:46.031666 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:46.031579 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8md2" event={"ID":"7cc81f17-d52c-4edb-9f33-de2f37c80b54","Type":"ContainerStarted","Data":"fdb6af5038c950e41b3fa37c07314e13f73a8e3aeab135b8b80828659d3b8ee5"} Nov 03 22:22:46.191102 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:46.191060 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rcrd8"] Nov 03 22:22:46.232818 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:46.232761 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde375da1_1b02_4796_ad49_2062bc5bc74b.slice/crio-a82bb85ad8e9820cf33b187fcf2362d47124d69b568ee82bbbd7b851de86f851 WatchSource:0}: Error finding container a82bb85ad8e9820cf33b187fcf2362d47124d69b568ee82bbbd7b851de86f851: Status 404 returned error can't find the container with id a82bb85ad8e9820cf33b187fcf2362d47124d69b568ee82bbbd7b851de86f851 Nov 03 22:22:46.605755 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:46.605728 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:22:46.605982 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:46.605968 4898 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:22:47.035464 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:47.035395 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcrd8" event={"ID":"de375da1-1b02-4796-ad49-2062bc5bc74b","Type":"ContainerStarted","Data":"a82bb85ad8e9820cf33b187fcf2362d47124d69b568ee82bbbd7b851de86f851"} Nov 03 22:22:47.332120 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:47.332090 4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-173.us-west-2.compute.internal" event="NodeReady" Nov 03 22:22:48.041288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:48.041202 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcrd8" event={"ID":"de375da1-1b02-4796-ad49-2062bc5bc74b","Type":"ContainerStarted","Data":"4f0530d871338da3f4c57f3023c7ed7a206482e9fbdd4f69900f7bd81de5393f"} Nov 03 22:22:48.041741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:48.041294 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcrd8" event={"ID":"de375da1-1b02-4796-ad49-2062bc5bc74b","Type":"ContainerStarted","Data":"58966a0290514e58b1e86bfada747eada4805fc9f07f16fc83174b23437965dd"} Nov 03 22:22:49.833668 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:22:49.833634 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:22:50.899570 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:50.899534 4898 scope.go:122] "RemoveContainer" containerID="1db13483c24c13a859ac1ff207be20a6f060c42aa749aed73c33867b1c800169" Nov 03 22:22:51.053534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:51.053501 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:22:51.053713 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:51.053585 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" event={"ID":"e3a1808a-9488-4d49-8028-4860660c0c9d","Type":"ContainerStarted","Data":"a2c6cd98d6469d2cd74926f3a0a12c06746e611eaea7694cfb7f78e4fd2cfa94"} Nov 03 22:22:51.054252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:51.054218 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:51.055649 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:51.055616 4898 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-thjpw container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body= Nov 03 22:22:51.055777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:51.055716 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" podUID="e3a1808a-9488-4d49-8028-4860660c0c9d" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" Nov 03 22:22:52.062012 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:52.061980 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-thjpw" Nov 03 22:22:53.827904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.827856 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:53.828265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.827923 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:53.828265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.828097 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:53.828265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.828143 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:53.828265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.828227 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:53.828950 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.828927 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700b9d15-342a-46d1-b94e-6b44aae2d9b5-service-ca-bundle\") pod \"router-default-74d647c79-nsnqv\" (UID: \"700b9d15-342a-46d1-b94e-6b44aae2d9b5\") " pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:53.830374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.830349 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"image-registry-5d9b64b6dc-xqdzm\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:53.830524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.830503 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"image-registry-58d49cb494-tnd66\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:53.830613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.830598 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e1c673-629b-4840-9f7a-aa64777cf32a-metrics-tls\") pod \"dns-default-jmjbp\" (UID: \"68e1c673-629b-4840-9f7a-aa64777cf32a\") " pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:53.830721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.830704 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ebd614-43bd-479e-8e95-c33b2c93361c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kb8lw\" (UID: \"64ebd614-43bd-479e-8e95-c33b2c93361c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:53.910835 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.910784 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" Nov 03 22:22:53.912979 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.912958 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:53.913071 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.913050 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:53.918850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.918831 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:53.920180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:53.920162 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:54.294453 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:54.294384 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw"] Nov 03 22:22:54.347712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:54.347685 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74d647c79-nsnqv"] Nov 03 22:22:54.384927 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:54.384897 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700b9d15_342a_46d1_b94e_6b44aae2d9b5.slice/crio-e06af51ec5f42bc2ca73b22be7e9bf891445b3e72d71e6300f2d2db01ba3a860 WatchSource:0}: Error finding container e06af51ec5f42bc2ca73b22be7e9bf891445b3e72d71e6300f2d2db01ba3a860: Status 404 returned error can't find the container with id e06af51ec5f42bc2ca73b22be7e9bf891445b3e72d71e6300f2d2db01ba3a860 Nov 03 22:22:54.395714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:54.395657 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jmjbp"] Nov 03 22:22:54.398472 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:54.398413 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e1c673_629b_4840_9f7a_aa64777cf32a.slice/crio-5d35beb641451f4897f3d277dcd42654e1a8d647c77671164c8fea5317edd4ca WatchSource:0}: Error finding container 5d35beb641451f4897f3d277dcd42654e1a8d647c77671164c8fea5317edd4ca: Status 404 returned error can't find the container with id 5d35beb641451f4897f3d277dcd42654e1a8d647c77671164c8fea5317edd4ca Nov 03 22:22:54.398519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:54.398501 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d49cb494-tnd66"] Nov 03 22:22:54.401128 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:54.401102 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e5df88_99b0_4026_8ae2_b71752948379.slice/crio-08403b0ff283e1f7cb6269f91d929d007fbb256d5eee6667773aca2700c04aa9 WatchSource:0}: Error finding container 08403b0ff283e1f7cb6269f91d929d007fbb256d5eee6667773aca2700c04aa9: Status 404 returned error can't find the container with id 08403b0ff283e1f7cb6269f91d929d007fbb256d5eee6667773aca2700c04aa9 Nov 03 22:22:54.402189 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:54.402014 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d9b64b6dc-xqdzm"] Nov 03 22:22:54.412714 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:54.412690 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b569914_42dd_4a2f_bd2c_63b37b3c47c9.slice/crio-c5c3a3987f4072245d797d912cdd6c0998ea1c52a9095003fe999633615f6472 WatchSource:0}: Error finding container c5c3a3987f4072245d797d912cdd6c0998ea1c52a9095003fe999633615f6472: Status 404 returned error can't find the container with id c5c3a3987f4072245d797d912cdd6c0998ea1c52a9095003fe999633615f6472 Nov 03 22:22:55.070486 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.070438 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74d647c79-nsnqv" event={"ID":"700b9d15-342a-46d1-b94e-6b44aae2d9b5","Type":"ContainerStarted","Data":"386164cff915b0fdd286a6e0c382f6165d3d50f7c4c27afc1d97987e89846018"} Nov 03 22:22:55.070981 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.070512 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74d647c79-nsnqv" event={"ID":"700b9d15-342a-46d1-b94e-6b44aae2d9b5","Type":"ContainerStarted","Data":"e06af51ec5f42bc2ca73b22be7e9bf891445b3e72d71e6300f2d2db01ba3a860"} Nov 03 22:22:55.071853 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.071817 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" event={"ID":"64ebd614-43bd-479e-8e95-c33b2c93361c","Type":"ContainerStarted","Data":"579a3a002f5689ccbf12e00d2a1163ac7567897355784cdaea5f0ab4bd88a877"} Nov 03 22:22:55.073490 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.073465 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" event={"ID":"07e5df88-99b0-4026-8ae2-b71752948379","Type":"ContainerStarted","Data":"45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb"} Nov 03 22:22:55.073606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.073505 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" event={"ID":"07e5df88-99b0-4026-8ae2-b71752948379","Type":"ContainerStarted","Data":"08403b0ff283e1f7cb6269f91d929d007fbb256d5eee6667773aca2700c04aa9"} Nov 03 22:22:55.073606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.073593 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:22:55.075245 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.075221 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" event={"ID":"3b569914-42dd-4a2f-bd2c-63b37b3c47c9","Type":"ContainerStarted","Data":"70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e"} Nov 03 22:22:55.075338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.075261 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" event={"ID":"3b569914-42dd-4a2f-bd2c-63b37b3c47c9","Type":"ContainerStarted","Data":"c5c3a3987f4072245d797d912cdd6c0998ea1c52a9095003fe999633615f6472"} Nov 03 22:22:55.075338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.075286 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:22:55.076502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.076483 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmjbp" event={"ID":"68e1c673-629b-4840-9f7a-aa64777cf32a","Type":"ContainerStarted","Data":"5d35beb641451f4897f3d277dcd42654e1a8d647c77671164c8fea5317edd4ca"} Nov 03 22:22:55.866459 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.866420 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pvpqf" Nov 03 22:22:55.913504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.913473 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:55.916915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:55.916890 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:56.082361 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:56.082327 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:56.083411 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:56.083389 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74d647c79-nsnqv" Nov 03 22:22:57.086895 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:57.086821 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmjbp" event={"ID":"68e1c673-629b-4840-9f7a-aa64777cf32a","Type":"ContainerStarted","Data":"5d238fa24c3e10ae007cd269aecc698c8429d3f4e2a15493c3560aa4c6287b90"} Nov 03 22:22:57.087361 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:57.086957 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmjbp" event={"ID":"68e1c673-629b-4840-9f7a-aa64777cf32a","Type":"ContainerStarted","Data":"f43f609269b61778904fad34b24bb366036de12950d93632bf5caf00a1757c96"} Nov 03 22:22:57.087361 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:57.086992 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jmjbp" Nov 03 22:22:57.088629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:57.088601 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" event={"ID":"64ebd614-43bd-479e-8e95-c33b2c93361c","Type":"ContainerStarted","Data":"81cfc66c2827807904852b0b1dd5a7228ed68133a2b6f0e30bd27a5f41a6ee7a"} Nov 03 22:22:57.088748 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:57.088652 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kb8lw" event={"ID":"64ebd614-43bd-479e-8e95-c33b2c93361c","Type":"ContainerStarted","Data":"a6d45b4b57f9e81d6e4555107a972f685ce88f8c8d301dbef4a49012b54ef288"} Nov 03 22:22:58.706574 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.706544 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk"] Nov 03 22:22:58.711163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.711140 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-8w6hx"] Nov 03 22:22:58.711285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.711263 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" Nov 03 22:22:58.714712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.714690 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:58.756422 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.756393 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d9b64b6dc-xqdzm"] Nov 03 22:22:58.767862 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.767830 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-g5mdn\"" Nov 03 22:22:58.768022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.767927 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 22:22:58.774418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.774398 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:22:58.776455 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.776432 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fce14729-dea1-47e8-9d4c-c4aa3b54130b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wc6kk\" (UID: \"fce14729-dea1-47e8-9d4c-c4aa3b54130b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" Nov 03 22:22:58.780156 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.780135 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk"] Nov 03 22:22:58.798241 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.798211 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69bf847855-gx72z"] Nov 03 22:22:58.801545 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.801526 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.814361 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.814333 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8w6hx"] Nov 03 22:22:58.816155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.816136 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:22:58.849832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.849803 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69bf847855-gx72z"] Nov 03 22:22:58.865957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.865936 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-smgvb\"" Nov 03 22:22:58.877499 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877471 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71b7143a-0ecb-49d5-838c-d8f1825b02a0-ca-trust-extracted\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.877613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877513 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-registry-tls\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.877613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877551 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fce14729-dea1-47e8-9d4c-c4aa3b54130b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wc6kk\" (UID: \"fce14729-dea1-47e8-9d4c-c4aa3b54130b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" Nov 03 22:22:58.877725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877607 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71b7143a-0ecb-49d5-838c-d8f1825b02a0-installation-pull-secrets\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.877725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877638 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71b7143a-0ecb-49d5-838c-d8f1825b02a0-registry-certificates\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.877725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877669 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4t6f\" (UniqueName: \"kubernetes.io/projected/e159094f-1b3b-4b71-a376-0f6ab285d279-kube-api-access-f4t6f\") pod \"downloads-6dd789c8b7-8w6hx\" (UID: \"e159094f-1b3b-4b71-a376-0f6ab285d279\") " pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:58.877969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877762 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmnz\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-kube-api-access-pwmnz\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.877969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877806 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-bound-sa-token\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.877969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877882 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71b7143a-0ecb-49d5-838c-d8f1825b02a0-trusted-ca\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.877969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877910 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e159094f-1b3b-4b71-a376-0f6ab285d279-tmp\") pod \"downloads-6dd789c8b7-8w6hx\" (UID: \"e159094f-1b3b-4b71-a376-0f6ab285d279\") " pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:58.877969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.877945 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71b7143a-0ecb-49d5-838c-d8f1825b02a0-image-registry-private-configuration\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.880113 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.880094 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fce14729-dea1-47e8-9d4c-c4aa3b54130b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wc6kk\" (UID: \"fce14729-dea1-47e8-9d4c-c4aa3b54130b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" Nov 03 22:22:58.979219 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979141 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-registry-tls\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979219 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979197 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71b7143a-0ecb-49d5-838c-d8f1825b02a0-installation-pull-secrets\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979230 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71b7143a-0ecb-49d5-838c-d8f1825b02a0-registry-certificates\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979258 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4t6f\" (UniqueName: \"kubernetes.io/projected/e159094f-1b3b-4b71-a376-0f6ab285d279-kube-api-access-f4t6f\") pod \"downloads-6dd789c8b7-8w6hx\" (UID: \"e159094f-1b3b-4b71-a376-0f6ab285d279\") " pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979294 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmnz\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-kube-api-access-pwmnz\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979329 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-bound-sa-token\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979378 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71b7143a-0ecb-49d5-838c-d8f1825b02a0-trusted-ca\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979429 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e159094f-1b3b-4b71-a376-0f6ab285d279-tmp\") pod \"downloads-6dd789c8b7-8w6hx\" (UID: \"e159094f-1b3b-4b71-a376-0f6ab285d279\") " pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979456 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71b7143a-0ecb-49d5-838c-d8f1825b02a0-image-registry-private-configuration\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979499 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71b7143a-0ecb-49d5-838c-d8f1825b02a0-ca-trust-extracted\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979952 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71b7143a-0ecb-49d5-838c-d8f1825b02a0-ca-trust-extracted\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.979987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.979970 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e159094f-1b3b-4b71-a376-0f6ab285d279-tmp\") pod \"downloads-6dd789c8b7-8w6hx\" (UID: \"e159094f-1b3b-4b71-a376-0f6ab285d279\") " pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:58.980278 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.980257 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71b7143a-0ecb-49d5-838c-d8f1825b02a0-registry-certificates\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.980410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.980393 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71b7143a-0ecb-49d5-838c-d8f1825b02a0-trusted-ca\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.981883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.981861 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71b7143a-0ecb-49d5-838c-d8f1825b02a0-installation-pull-secrets\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.982261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.982241 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-registry-tls\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:58.982299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:58.982241 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71b7143a-0ecb-49d5-838c-d8f1825b02a0-image-registry-private-configuration\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:59.024875 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.024852 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" Nov 03 22:22:59.042305 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.042279 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-bound-sa-token\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:59.042474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.042453 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmnz\" (UniqueName: \"kubernetes.io/projected/71b7143a-0ecb-49d5-838c-d8f1825b02a0-kube-api-access-pwmnz\") pod \"image-registry-69bf847855-gx72z\" (UID: \"71b7143a-0ecb-49d5-838c-d8f1825b02a0\") " pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:59.042736 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.042718 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4t6f\" (UniqueName: \"kubernetes.io/projected/e159094f-1b3b-4b71-a376-0f6ab285d279-kube-api-access-f4t6f\") pod \"downloads-6dd789c8b7-8w6hx\" (UID: \"e159094f-1b3b-4b71-a376-0f6ab285d279\") " pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:59.113330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.113297 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:22:59.330578 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.330540 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:22:59.397035 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.396945 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk"] Nov 03 22:22:59.416940 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:59.416909 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce14729_dea1_47e8_9d4c_c4aa3b54130b.slice/crio-4b5cc5613f96bc0e738db1a33b86f80ad3745f0c2112691f3cf6fe3f1af019bf WatchSource:0}: Error finding container 4b5cc5613f96bc0e738db1a33b86f80ad3745f0c2112691f3cf6fe3f1af019bf: Status 404 returned error can't find the container with id 4b5cc5613f96bc0e738db1a33b86f80ad3745f0c2112691f3cf6fe3f1af019bf Nov 03 22:22:59.487850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.487807 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69bf847855-gx72z"] Nov 03 22:22:59.489771 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:59.489751 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b7143a_0ecb_49d5_838c_d8f1825b02a0.slice/crio-b0969278e11d9717d37b15319dd264ab25dccc29e5376332766e5c3ba258aeda WatchSource:0}: Error finding container b0969278e11d9717d37b15319dd264ab25dccc29e5376332766e5c3ba258aeda: Status 404 returned error can't find the container with id b0969278e11d9717d37b15319dd264ab25dccc29e5376332766e5c3ba258aeda Nov 03 22:22:59.702730 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:22:59.702601 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8w6hx"] Nov 03 22:22:59.705494 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:22:59.705461 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode159094f_1b3b_4b71_a376_0f6ab285d279.slice/crio-0fed3b5a4401bc621320e90676113f18f594cf0c82c0027ce5c6c9523e9be59c WatchSource:0}: Error finding container 0fed3b5a4401bc621320e90676113f18f594cf0c82c0027ce5c6c9523e9be59c: Status 404 returned error can't find the container with id 0fed3b5a4401bc621320e90676113f18f594cf0c82c0027ce5c6c9523e9be59c Nov 03 22:23:00.108098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:00.108063 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69bf847855-gx72z" event={"ID":"71b7143a-0ecb-49d5-838c-d8f1825b02a0","Type":"ContainerStarted","Data":"98c6246c2d8e402b25497d38e73c8cef57e2111fe77ce1d1f64a89678e38a8b7"} Nov 03 22:23:00.108556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:00.108119 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69bf847855-gx72z" event={"ID":"71b7143a-0ecb-49d5-838c-d8f1825b02a0","Type":"ContainerStarted","Data":"b0969278e11d9717d37b15319dd264ab25dccc29e5376332766e5c3ba258aeda"} Nov 03 22:23:00.108556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:00.108214 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:23:00.109389 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:00.109354 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8w6hx" event={"ID":"e159094f-1b3b-4b71-a376-0f6ab285d279","Type":"ContainerStarted","Data":"0fed3b5a4401bc621320e90676113f18f594cf0c82c0027ce5c6c9523e9be59c"} Nov 03 22:23:00.110503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:00.110478 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" event={"ID":"fce14729-dea1-47e8-9d4c-c4aa3b54130b","Type":"ContainerStarted","Data":"4b5cc5613f96bc0e738db1a33b86f80ad3745f0c2112691f3cf6fe3f1af019bf"} Nov 03 22:23:00.231689 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:00.231633 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69bf847855-gx72z" podStartSLOduration=2.231618428 podStartE2EDuration="2.231618428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:00.230517454 +0000 UTC m=+23.443286533" watchObservedRunningTime="2026-11-03 22:23:00.231618428 +0000 UTC m=+23.444387481" Nov 03 22:23:02.122559 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:02.122518 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" event={"ID":"fce14729-dea1-47e8-9d4c-c4aa3b54130b","Type":"ContainerStarted","Data":"2b9363f5bba2d9a80dbc2dc6162fd96e94690c7c9f2984a3252ac0ef51f68e7c"} Nov 03 22:23:02.123050 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:02.122727 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" Nov 03 22:23:02.128074 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:02.128052 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" Nov 03 22:23:02.238274 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:02.238211 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wc6kk" podStartSLOduration=2.3445306439999998 podStartE2EDuration="4.238196053s" totalImagesPullingTime="1.893665409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:02.237746869 +0000 UTC m=+25.450515943" watchObservedRunningTime="2026-11-03 22:23:02.238196053 +0000 UTC m=+25.450965104" Nov 03 22:23:03.733206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.733171 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p"] Nov 03 22:23:03.763125 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.763080 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:03.798712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.798674 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p"] Nov 03 22:23:03.842123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.842085 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 22:23:03.866364 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.866320 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 22:23:03.866364 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.866353 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:23:03.892346 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.892319 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d6zhf\"" Nov 03 22:23:03.928489 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.928456 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45e72fcb-4f35-48b5-8d07-efd0bda7809a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:03.928489 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.928494 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45e72fcb-4f35-48b5-8d07-efd0bda7809a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:03.928712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.928619 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45e72fcb-4f35-48b5-8d07-efd0bda7809a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:03.928712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:03.928684 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvxm\" (UniqueName: \"kubernetes.io/projected/45e72fcb-4f35-48b5-8d07-efd0bda7809a-kube-api-access-hpvxm\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.029532 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.029447 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45e72fcb-4f35-48b5-8d07-efd0bda7809a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.029532 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.029491 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45e72fcb-4f35-48b5-8d07-efd0bda7809a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.029742 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.029534 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45e72fcb-4f35-48b5-8d07-efd0bda7809a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.029742 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.029555 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvxm\" (UniqueName: \"kubernetes.io/projected/45e72fcb-4f35-48b5-8d07-efd0bda7809a-kube-api-access-hpvxm\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.030581 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.030554 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45e72fcb-4f35-48b5-8d07-efd0bda7809a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.032765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.032686 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45e72fcb-4f35-48b5-8d07-efd0bda7809a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.032906 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.032883 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45e72fcb-4f35-48b5-8d07-efd0bda7809a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.091148 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.091111 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvxm\" (UniqueName: \"kubernetes.io/projected/45e72fcb-4f35-48b5-8d07-efd0bda7809a-kube-api-access-hpvxm\") pod \"prometheus-operator-9cd89b74c-kqz7p\" (UID: \"45e72fcb-4f35-48b5-8d07-efd0bda7809a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.117010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.116981 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" Nov 03 22:23:04.517161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:04.517126 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p"] Nov 03 22:23:04.522729 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:23:04.522703 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e72fcb_4f35_48b5_8d07_efd0bda7809a.slice/crio-2ba71169414326b040095632ed10df0aa2e7ad442d39dc22328351d160917407 WatchSource:0}: Error finding container 2ba71169414326b040095632ed10df0aa2e7ad442d39dc22328351d160917407: Status 404 returned error can't find the container with id 2ba71169414326b040095632ed10df0aa2e7ad442d39dc22328351d160917407 Nov 03 22:23:05.140412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:05.140319 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" event={"ID":"45e72fcb-4f35-48b5-8d07-efd0bda7809a","Type":"ContainerStarted","Data":"2ba71169414326b040095632ed10df0aa2e7ad442d39dc22328351d160917407"} Nov 03 22:23:06.497328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:06.497295 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhx44" Nov 03 22:23:07.096504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:07.096438 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jmjbp" Nov 03 22:23:07.152727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:07.152694 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" event={"ID":"45e72fcb-4f35-48b5-8d07-efd0bda7809a","Type":"ContainerStarted","Data":"9d549113f4e39b8802a86d0095a008cca3224de7d6bd888414409105f76c4b1e"} Nov 03 22:23:07.152926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:07.152751 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" event={"ID":"45e72fcb-4f35-48b5-8d07-efd0bda7809a","Type":"ContainerStarted","Data":"ed730435b81e2847865b107498a3a3cd66300b8a5a2bc99e05328967f60f1071"} Nov 03 22:23:07.332796 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:07.332724 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kqz7p" podStartSLOduration=2.013413369 podStartE2EDuration="4.332708048s" totalImagesPullingTime="2.319294679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:07.332488668 +0000 UTC m=+30.545257745" watchObservedRunningTime="2026-11-03 22:23:07.332708048 +0000 UTC m=+30.545477110" Nov 03 22:23:10.289149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.289114 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vchhc"] Nov 03 22:23:10.317253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.317225 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.370352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.370319 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:23:10.423804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.423756 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:23:10.423981 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.423871 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:23:10.423981 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.423913 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2kdj\"" Nov 03 22:23:10.491516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491469 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-tls\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.491516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491519 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-accelerators-collector-config\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.491738 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491635 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-root\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.491738 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491667 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvpz\" (UniqueName: \"kubernetes.io/projected/037aef30-2cc0-42c0-83bc-b685b2409e60-kube-api-access-htvpz\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.491738 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491689 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/037aef30-2cc0-42c0-83bc-b685b2409e60-metrics-client-ca\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.491906 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491758 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-sys\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.491950 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491926 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-wtmp\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.492012 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.491995 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.492064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.492045 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-textfile\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.592834 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.592774 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-tls\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.592847 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-accelerators-collector-config\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.592884 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-root\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.592904 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvpz\" (UniqueName: \"kubernetes.io/projected/037aef30-2cc0-42c0-83bc-b685b2409e60-kube-api-access-htvpz\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.592926 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/037aef30-2cc0-42c0-83bc-b685b2409e60-metrics-client-ca\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.592955 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-sys\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.593002 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-wtmp\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593335 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.593061 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593335 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.593088 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-textfile\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.593439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.593397 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-textfile\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.594080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.594057 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-root\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.594186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.594171 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-wtmp\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.594235 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.594206 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/037aef30-2cc0-42c0-83bc-b685b2409e60-sys\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.594366 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.594342 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-accelerators-collector-config\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.594750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.594729 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/037aef30-2cc0-42c0-83bc-b685b2409e60-metrics-client-ca\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.596617 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.596595 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.596731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.596661 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/037aef30-2cc0-42c0-83bc-b685b2409e60-node-exporter-tls\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.666363 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.666333 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvpz\" (UniqueName: \"kubernetes.io/projected/037aef30-2cc0-42c0-83bc-b685b2409e60-kube-api-access-htvpz\") pod \"node-exporter-vchhc\" (UID: \"037aef30-2cc0-42c0-83bc-b685b2409e60\") " pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.674213 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:10.674190 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vchhc" Nov 03 22:23:10.705593 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:23:10.705566 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037aef30_2cc0_42c0_83bc_b685b2409e60.slice/crio-da8f7034c7aa5f9fb1f3f2a76af7debc4b5af147f21019b49e7ac03223296ea9 WatchSource:0}: Error finding container da8f7034c7aa5f9fb1f3f2a76af7debc4b5af147f21019b49e7ac03223296ea9: Status 404 returned error can't find the container with id da8f7034c7aa5f9fb1f3f2a76af7debc4b5af147f21019b49e7ac03223296ea9 Nov 03 22:23:11.174663 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:11.174616 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vchhc" event={"ID":"037aef30-2cc0-42c0-83bc-b685b2409e60","Type":"ContainerStarted","Data":"da8f7034c7aa5f9fb1f3f2a76af7debc4b5af147f21019b49e7ac03223296ea9"} Nov 03 22:23:12.087108 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:12.086388 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mzwbr" Nov 03 22:23:13.186999 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:13.186958 4898 generic.go:356] "Generic (PLEG): container finished" podID="037aef30-2cc0-42c0-83bc-b685b2409e60" containerID="9476647f26c8acf713d76eb206f3514d4b199eeec08083bbef06dc7062505bb7" exitCode=0 Nov 03 22:23:13.187431 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:13.187041 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vchhc" event={"ID":"037aef30-2cc0-42c0-83bc-b685b2409e60","Type":"ContainerDied","Data":"9476647f26c8acf713d76eb206f3514d4b199eeec08083bbef06dc7062505bb7"} Nov 03 22:23:13.924294 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:13.924260 4898 patch_prober.go:29] interesting pod/image-registry-58d49cb494-tnd66 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 22:23:13.924441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:13.924312 4898 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" podUID="07e5df88-99b0-4026-8ae2-b71752948379" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:23:14.194730 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:14.194658 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vchhc" event={"ID":"037aef30-2cc0-42c0-83bc-b685b2409e60","Type":"ContainerStarted","Data":"4ae80ff111956a8b297dd2b1ebe794dfd29e80e52e06abb8e7df30883d77c649"} Nov 03 22:23:14.194730 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:14.194707 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vchhc" event={"ID":"037aef30-2cc0-42c0-83bc-b685b2409e60","Type":"ContainerStarted","Data":"71a19d2e0fa4b62cbfd9084913023888597d386f3045633249f3610cc36c7443"} Nov 03 22:23:14.316683 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:14.316624 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vchhc" podStartSLOduration=2.118658796 podStartE2EDuration="4.316605052s" totalImagesPullingTime="2.100171146s" totalInitContainerRuntime="97.77511ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:14.315659412 +0000 UTC m=+37.528428467" watchObservedRunningTime="2026-11-03 22:23:14.316605052 +0000 UTC m=+37.529374107" Nov 03 22:23:16.086734 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:16.086694 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:23:17.209902 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:17.209779 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8w6hx" event={"ID":"e159094f-1b3b-4b71-a376-0f6ab285d279","Type":"ContainerStarted","Data":"7f874f6e9f00b93fae9488a0029d4fc6341a4375c9a413a4d2bc627ae496291a"} Nov 03 22:23:17.210383 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:17.210027 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:23:17.231472 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:17.231443 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-8w6hx" Nov 03 22:23:17.378282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:17.378224 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-8w6hx" podStartSLOduration=2.772813154 podStartE2EDuration="19.378207936s" totalImagesPullingTime="16.605394782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:17.327817678 +0000 UTC m=+40.540586733" watchObservedRunningTime="2026-11-03 22:23:17.378207936 +0000 UTC m=+40.590976987" Nov 03 22:23:18.762737 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:18.762707 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:23:19.839039 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:23:19.839000 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:23:21.121040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:21.121004 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69bf847855-gx72z" Nov 03 22:23:21.372668 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:21.372597 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d49cb494-tnd66"] Nov 03 22:23:23.776296 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:23.776249 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" podUID="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" containerName="registry" containerID="cri-o://70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e" gracePeriod=30 Nov 03 22:23:24.088059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.088031 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:23:24.120174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120134 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.120357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120189 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.120357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120237 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.120357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120272 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.120357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120309 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.120357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120353 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.120635 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120410 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.120635 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.120456 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\") pod \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\" (UID: \"3b569914-42dd-4a2f-bd2c-63b37b3c47c9\") " Nov 03 22:23:24.121155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.121117 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:24.121966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.121924 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:24.124299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.124253 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:24.124648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.124616 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "kube-api-access-knc9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:24.125474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.125434 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:24.126076 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.126047 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:24.126650 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.126624 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:24.129341 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.129314 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted" pod "3b569914-42dd-4a2f-bd2c-63b37b3c47c9" (UID: "3b569914-42dd-4a2f-bd2c-63b37b3c47c9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:24.221944 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.221909 4898 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-ca-trust-extracted\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.221944 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.221942 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-kube-api-access-knc9r\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.222168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.221955 4898 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-bound-sa-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.222168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.221965 4898 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-installation-pull-secrets\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.222168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.221977 4898 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-image-registry-private-configuration\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.222168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.221987 4898 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.222168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.221999 4898 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-registry-certificates\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.222168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.222011 4898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b569914-42dd-4a2f-bd2c-63b37b3c47c9-trusted-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:24.247864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.247821 4898 generic.go:356] "Generic (PLEG): container finished" podID="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" containerID="70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e" exitCode=0 Nov 03 22:23:24.248037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.247871 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" Nov 03 22:23:24.248037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.247873 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" event={"ID":"3b569914-42dd-4a2f-bd2c-63b37b3c47c9","Type":"ContainerDied","Data":"70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e"} Nov 03 22:23:24.248037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.248014 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d9b64b6dc-xqdzm" event={"ID":"3b569914-42dd-4a2f-bd2c-63b37b3c47c9","Type":"ContainerDied","Data":"c5c3a3987f4072245d797d912cdd6c0998ea1c52a9095003fe999633615f6472"} Nov 03 22:23:24.248276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.248042 4898 scope.go:122] "RemoveContainer" containerID="70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e" Nov 03 22:23:24.258955 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.258928 4898 scope.go:122] "RemoveContainer" containerID="70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e" Nov 03 22:23:24.259306 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:23:24.259282 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e\": container with ID starting with 70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e not found: ID does not exist" containerID="70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e" Nov 03 22:23:24.259410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.259318 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e"} err="failed to get container status \"70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e\": rpc error: code = NotFound desc = could not find container \"70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e\": container with ID starting with 70218ad02911246c7599e73830b319725f3140ca057e30f33ee1833c3182217e not found: ID does not exist" Nov 03 22:23:24.424946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.424914 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d9b64b6dc-xqdzm"] Nov 03 22:23:24.429242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.429213 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d9b64b6dc-xqdzm"] Nov 03 22:23:24.907097 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:24.907062 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" path="/var/lib/kubelet/pods/3b569914-42dd-4a2f-bd2c-63b37b3c47c9/volumes" Nov 03 22:23:46.396391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.396321 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" podUID="07e5df88-99b0-4026-8ae2-b71752948379" containerName="registry" containerID="cri-o://45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb" gracePeriod=30 Nov 03 22:23:46.692173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.692144 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:23:46.813837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.813803 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.813837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.813851 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.814101 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.813872 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.814101 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.813901 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.814209 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.814092 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.814209 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.814157 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.814321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.814210 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.814321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.814258 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\") pod \"07e5df88-99b0-4026-8ae2-b71752948379\" (UID: \"07e5df88-99b0-4026-8ae2-b71752948379\") " Nov 03 22:23:46.814428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.814361 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:46.814699 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.814672 4898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-trusted-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:46.815290 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.815256 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:46.816611 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.816580 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "kube-api-access-ghp7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:46.816722 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.816655 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:46.816813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.816721 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:46.816813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.816736 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:46.816893 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.816848 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:46.821314 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.821288 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted" pod "07e5df88-99b0-4026-8ae2-b71752948379" (UID: "07e5df88-99b0-4026-8ae2-b71752948379"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:46.915052 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.915021 4898 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e5df88-99b0-4026-8ae2-b71752948379-ca-trust-extracted\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:46.915174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.915055 4898 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e5df88-99b0-4026-8ae2-b71752948379-registry-certificates\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:46.915174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.915071 4898 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-installation-pull-secrets\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:46.915174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.915085 4898 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-bound-sa-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:46.915174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.915099 4898 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e5df88-99b0-4026-8ae2-b71752948379-image-registry-private-configuration\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:46.915174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.915113 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghp7z\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-kube-api-access-ghp7z\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:46.915174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:46.915126 4898 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e5df88-99b0-4026-8ae2-b71752948379-registry-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:47.342521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.342482 4898 generic.go:356] "Generic (PLEG): container finished" podID="07e5df88-99b0-4026-8ae2-b71752948379" containerID="45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb" exitCode=0 Nov 03 22:23:47.342705 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.342537 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" Nov 03 22:23:47.342705 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.342566 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" event={"ID":"07e5df88-99b0-4026-8ae2-b71752948379","Type":"ContainerDied","Data":"45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb"} Nov 03 22:23:47.342705 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.342618 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d49cb494-tnd66" event={"ID":"07e5df88-99b0-4026-8ae2-b71752948379","Type":"ContainerDied","Data":"08403b0ff283e1f7cb6269f91d929d007fbb256d5eee6667773aca2700c04aa9"} Nov 03 22:23:47.342705 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.342638 4898 scope.go:122] "RemoveContainer" containerID="45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb" Nov 03 22:23:47.350960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.350940 4898 scope.go:122] "RemoveContainer" containerID="45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb" Nov 03 22:23:47.351220 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:23:47.351202 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb\": container with ID starting with 45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb not found: ID does not exist" containerID="45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb" Nov 03 22:23:47.351270 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.351230 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb"} err="failed to get container status \"45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb\": rpc error: code = NotFound desc = could not find container \"45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb\": container with ID starting with 45382405e1f790909c8c22203ff7f78eeea1d608d7130d4fa127478dce2ccefb not found: ID does not exist" Nov 03 22:23:47.509003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.508970 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d49cb494-tnd66"] Nov 03 22:23:47.559195 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:47.559158 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58d49cb494-tnd66"] Nov 03 22:23:48.904221 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:48.904189 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e5df88-99b0-4026-8ae2-b71752948379" path="/var/lib/kubelet/pods/07e5df88-99b0-4026-8ae2-b71752948379/volumes" Nov 03 22:23:49.833461 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:23:49.833422 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:23:54.373893 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:54.373859 4898 generic.go:356] "Generic (PLEG): container finished" podID="d5210f23-6d88-413f-b0b5-c1b938713ca9" containerID="a844ae340dbed8549b157f1dbacbfbc254d0e95bf71f727d0da9a6792ff74020" exitCode=0 Nov 03 22:23:54.374272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:54.373932 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z5lhd" event={"ID":"d5210f23-6d88-413f-b0b5-c1b938713ca9","Type":"ContainerDied","Data":"a844ae340dbed8549b157f1dbacbfbc254d0e95bf71f727d0da9a6792ff74020"} Nov 03 22:23:54.374272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:54.374255 4898 scope.go:122] "RemoveContainer" containerID="a844ae340dbed8549b157f1dbacbfbc254d0e95bf71f727d0da9a6792ff74020" Nov 03 22:23:55.378993 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.378960 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z5lhd" event={"ID":"d5210f23-6d88-413f-b0b5-c1b938713ca9","Type":"ContainerStarted","Data":"a69f7bf02a50f1f6c0ae36182ffaf9d30c07d557cc48c85b132adbb1a30814cb"} Nov 03 22:23:55.380443 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.380417 4898 generic.go:356] "Generic (PLEG): container finished" podID="65f4d5dd-a512-425a-91b9-de7749f091c6" containerID="6e7a9a5fff1a9b5e3b191f58c3008c5b45b77ccb096a44b3aec96b0f450df434" exitCode=0 Nov 03 22:23:55.380547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.380454 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" event={"ID":"65f4d5dd-a512-425a-91b9-de7749f091c6","Type":"ContainerDied","Data":"6e7a9a5fff1a9b5e3b191f58c3008c5b45b77ccb096a44b3aec96b0f450df434"} Nov 03 22:23:55.380722 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.380711 4898 scope.go:122] "RemoveContainer" containerID="6e7a9a5fff1a9b5e3b191f58c3008c5b45b77ccb096a44b3aec96b0f450df434" Nov 03 22:23:55.483515 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.483481 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fnl7t-pdbr9"] Nov 03 22:23:55.483722 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.483700 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" podUID="e0322b8e-5e35-4e52-b5dd-42f32573354f" containerName="insights-gathering" containerID="cri-o://990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1" gracePeriod=30 Nov 03 22:23:55.786856 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.786827 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:23:55.890562 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.890530 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\") pod \"e0322b8e-5e35-4e52-b5dd-42f32573354f\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " Nov 03 22:23:55.890740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.890578 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\" (UniqueName: \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\") pod \"e0322b8e-5e35-4e52-b5dd-42f32573354f\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " Nov 03 22:23:55.890740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.890601 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\") pod \"e0322b8e-5e35-4e52-b5dd-42f32573354f\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " Nov 03 22:23:55.890740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.890632 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\") pod \"e0322b8e-5e35-4e52-b5dd-42f32573354f\" (UID: \"e0322b8e-5e35-4e52-b5dd-42f32573354f\") " Nov 03 22:23:55.891020 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.890992 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path" pod "e0322b8e-5e35-4e52-b5dd-42f32573354f" (UID: "e0322b8e-5e35-4e52-b5dd-42f32573354f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:55.891084 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.891058 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle" pod "e0322b8e-5e35-4e52-b5dd-42f32573354f" (UID: "e0322b8e-5e35-4e52-b5dd-42f32573354f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:55.891142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.891085 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle" pod "e0322b8e-5e35-4e52-b5dd-42f32573354f" (UID: "e0322b8e-5e35-4e52-b5dd-42f32573354f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:55.893114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.893093 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b" pod "e0322b8e-5e35-4e52-b5dd-42f32573354f" (UID: "e0322b8e-5e35-4e52-b5dd-42f32573354f"). InnerVolumeSpecName "kube-api-access-x762b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:55.991648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.991564 4898 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0322b8e-5e35-4e52-b5dd-42f32573354f-archives-path\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:55.991648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.991597 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x762b\" (UniqueName: \"kubernetes.io/projected/e0322b8e-5e35-4e52-b5dd-42f32573354f-kube-api-access-x762b\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:55.991648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.991612 4898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-trusted-ca-bundle\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:55.991648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:55.991626 4898 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0322b8e-5e35-4e52-b5dd-42f32573354f-service-ca-bundle\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:56.385978 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.385938 4898 generic.go:356] "Generic (PLEG): container finished" podID="e0322b8e-5e35-4e52-b5dd-42f32573354f" containerID="990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1" exitCode=2 Nov 03 22:23:56.386424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.385987 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" Nov 03 22:23:56.386424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.386021 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" event={"ID":"e0322b8e-5e35-4e52-b5dd-42f32573354f","Type":"ContainerDied","Data":"990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1"} Nov 03 22:23:56.386424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.386082 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fnl7t-pdbr9" event={"ID":"e0322b8e-5e35-4e52-b5dd-42f32573354f","Type":"ContainerDied","Data":"2a3cfb6a46c5b5d03bb879c07e7f44f45f98a62dce556bcfdb3cd279cf7058fe"} Nov 03 22:23:56.386424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.386112 4898 scope.go:122] "RemoveContainer" containerID="990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1" Nov 03 22:23:56.388227 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.388204 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7khh5" event={"ID":"65f4d5dd-a512-425a-91b9-de7749f091c6","Type":"ContainerStarted","Data":"74b65283cefc89923a9ffa3af49008c2829ca52a6712285f3eac70062020591b"} Nov 03 22:23:56.395745 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.395390 4898 scope.go:122] "RemoveContainer" containerID="990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1" Nov 03 22:23:56.395745 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:23:56.395675 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1\": container with ID starting with 990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1 not found: ID does not exist" containerID="990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1" Nov 03 22:23:56.395745 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.395701 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1"} err="failed to get container status \"990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1\": rpc error: code = NotFound desc = could not find container \"990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1\": container with ID starting with 990c7001c4ae1ab7bce079ea6f7430afecf02a38709f7700d2dab084c73981c1 not found: ID does not exist" Nov 03 22:23:56.667257 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.667171 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fnl7t-pdbr9"] Nov 03 22:23:56.671062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.671035 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-fnl7t-pdbr9"] Nov 03 22:23:56.908023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:23:56.907979 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0322b8e-5e35-4e52-b5dd-42f32573354f" path="/var/lib/kubelet/pods/e0322b8e-5e35-4e52-b5dd-42f32573354f/volumes" Nov 03 22:24:01.414621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:24:01.414586 4898 generic.go:356] "Generic (PLEG): container finished" podID="4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34" containerID="d762dfe0c536292aa8e8a2de58744cdd600913e58b5205a7ded54c9829074d14" exitCode=0 Nov 03 22:24:01.415167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:24:01.414654 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" event={"ID":"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34","Type":"ContainerDied","Data":"d762dfe0c536292aa8e8a2de58744cdd600913e58b5205a7ded54c9829074d14"} Nov 03 22:24:01.415167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:24:01.415047 4898 scope.go:122] "RemoveContainer" containerID="d762dfe0c536292aa8e8a2de58744cdd600913e58b5205a7ded54c9829074d14" Nov 03 22:24:02.421312 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:24:02.421280 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gwbh6" event={"ID":"4ad3ef2b-1bf1-41ec-8848-a2ffd108fb34","Type":"ContainerStarted","Data":"b60f0a5e814d1f2116869e1630cb506465f97e171ad883951cfb94abf1b02b3d"} Nov 03 22:24:19.832593 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:24:19.832554 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:24:49.833398 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:24:49.833359 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:25:19.833104 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:25:19.833065 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:25:49.835753 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:25:49.835711 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:25:58.403748 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.403716 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv"] Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404072 4898 cpu_manager.go:414] "containerMap: removing container" podUID="e0322b8e-5e35-4e52-b5dd-42f32573354f" containerName="insights-gathering" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404083 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0322b8e-5e35-4e52-b5dd-42f32573354f" containerName="insights-gathering" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404097 4898 cpu_manager.go:414] "containerMap: removing container" podUID="07e5df88-99b0-4026-8ae2-b71752948379" containerName="registry" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404102 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e5df88-99b0-4026-8ae2-b71752948379" containerName="registry" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404112 4898 cpu_manager.go:414] "containerMap: removing container" podUID="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" containerName="registry" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404117 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" containerName="registry" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404178 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0322b8e-5e35-4e52-b5dd-42f32573354f" containerName="insights-gathering" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404189 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e5df88-99b0-4026-8ae2-b71752948379" containerName="registry" Nov 03 22:25:58.404250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.404195 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b569914-42dd-4a2f-bd2c-63b37b3c47c9" containerName="registry" Nov 03 22:25:58.406019 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.406003 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.459334 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.459251 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5vrs\"" Nov 03 22:25:58.459334 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.459251 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:25:58.510813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.510757 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv"] Nov 03 22:25:58.525235 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525199 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525398 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525251 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhjr\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-kube-api-access-mhhjr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525398 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525275 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525423 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525455 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525483 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525559 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525605 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a701c-59e5-49be-a92b-76cab43d4d90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.525767 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.525729 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626458 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626426 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a701c-59e5-49be-a92b-76cab43d4d90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626469 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626501 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626615 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhjr\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-kube-api-access-mhhjr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626667 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626729 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626759 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.626846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626812 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.627050 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626864 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.627050 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626915 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.627050 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.626960 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.627199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.627123 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.627199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.627154 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.627199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.627163 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a701c-59e5-49be-a92b-76cab43d4d90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.629096 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.629073 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.629461 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.629438 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.688569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.688492 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.688569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.688554 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhjr\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-kube-api-access-mhhjr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-d5brv\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:58.717263 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:58.717233 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:25:59.143250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:59.143223 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv"] Nov 03 22:25:59.145483 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:25:59.145454 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4a701c_59e5_49be_a92b_76cab43d4d90.slice/crio-90351c1aa952fc95dccdbc9c60a2d9788322d2ce0357170c3868299b2bead30f WatchSource:0}: Error finding container 90351c1aa952fc95dccdbc9c60a2d9788322d2ce0357170c3868299b2bead30f: Status 404 returned error can't find the container with id 90351c1aa952fc95dccdbc9c60a2d9788322d2ce0357170c3868299b2bead30f Nov 03 22:25:59.884507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:25:59.884472 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" event={"ID":"ff4a701c-59e5-49be-a92b-76cab43d4d90","Type":"ContainerStarted","Data":"90351c1aa952fc95dccdbc9c60a2d9788322d2ce0357170c3868299b2bead30f"} Nov 03 22:26:03.286664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:03.286633 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:26:03.286943 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:03.286703 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:26:03.286943 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:03.286736 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:26:03.909755 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:03.909712 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" event={"ID":"ff4a701c-59e5-49be-a92b-76cab43d4d90","Type":"ContainerStarted","Data":"4d7b86a45d0023aa6a293952300bac25ecdc62921c316170cb9f6057ff82b589"} Nov 03 22:26:04.081968 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:04.081920 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podStartSLOduration=1.943446692 podStartE2EDuration="6.081905863s" totalImagesPullingTime="4.138459171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:04.0810407 +0000 UTC m=+207.293809751" watchObservedRunningTime="2026-11-03 22:26:04.081905863 +0000 UTC m=+207.294674911" Nov 03 22:26:04.718462 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:04.718420 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:26:04.723057 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:04.723033 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:26:04.913672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:04.913637 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:26:04.914659 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:04.914641 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:26:19.835851 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:26:19.835761 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:26:31.209776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.209740 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp"] Nov 03 22:26:31.212391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.212373 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.294131 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.294096 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:26:31.300310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.300281 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.300450 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.300319 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54g4\" (UniqueName: \"kubernetes.io/projected/a7974b57-e611-42b8-b320-3c82f1146d06-kube-api-access-g54g4\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.300450 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.300400 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7974b57-e611-42b8-b320-3c82f1146d06-tmp\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.300450 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.300426 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.300586 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.300455 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-metrics-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.300586 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.300477 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7974b57-e611-42b8-b320-3c82f1146d06-manifests\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.316661 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.316642 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:26:31.316661 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.316661 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6xv29\"" Nov 03 22:26:31.316855 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.316642 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:26:31.316855 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.316781 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:26:31.316966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.316856 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:26:31.318585 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.318560 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp"] Nov 03 22:26:31.401922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.401895 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7974b57-e611-42b8-b320-3c82f1146d06-tmp\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.401922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.401924 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.402101 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.401951 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-metrics-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.402101 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.401970 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7974b57-e611-42b8-b320-3c82f1146d06-manifests\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.402101 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.401994 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.402246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.402110 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54g4\" (UniqueName: \"kubernetes.io/projected/a7974b57-e611-42b8-b320-3c82f1146d06-kube-api-access-g54g4\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.402466 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.402431 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7974b57-e611-42b8-b320-3c82f1146d06-manifests\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.404472 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.404431 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7974b57-e611-42b8-b320-3c82f1146d06-tmp\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.404577 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.404484 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.404651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.404633 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-metrics-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.404870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.404852 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7974b57-e611-42b8-b320-3c82f1146d06-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.463407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.463353 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54g4\" (UniqueName: \"kubernetes.io/projected/a7974b57-e611-42b8-b320-3c82f1146d06-kube-api-access-g54g4\") pod \"opendatahub-operator-controller-manager-8bc866f6-9gvrp\" (UID: \"a7974b57-e611-42b8-b320-3c82f1146d06\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:31.525315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:31.525291 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:32.016148 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:32.015984 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp"] Nov 03 22:26:32.019107 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:26:32.019081 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7974b57_e611_42b8_b320_3c82f1146d06.slice/crio-bae9a87221a88d76a7a1fe7f8b6470026c17407f8e0a55c04d3b50c4b1f0e22c WatchSource:0}: Error finding container bae9a87221a88d76a7a1fe7f8b6470026c17407f8e0a55c04d3b50c4b1f0e22c: Status 404 returned error can't find the container with id bae9a87221a88d76a7a1fe7f8b6470026c17407f8e0a55c04d3b50c4b1f0e22c Nov 03 22:26:33.025568 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:33.025535 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" event={"ID":"a7974b57-e611-42b8-b320-3c82f1146d06","Type":"ContainerStarted","Data":"bae9a87221a88d76a7a1fe7f8b6470026c17407f8e0a55c04d3b50c4b1f0e22c"} Nov 03 22:26:36.042122 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:36.042034 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" event={"ID":"a7974b57-e611-42b8-b320-3c82f1146d06","Type":"ContainerStarted","Data":"f1d401f75979a8b7352e0be203549f18ecaafe4ab45f19bc073373c5eed318a7"} Nov 03 22:26:37.047876 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:37.047840 4898 generic.go:356] "Generic (PLEG): container finished" podID="a7974b57-e611-42b8-b320-3c82f1146d06" containerID="f1d401f75979a8b7352e0be203549f18ecaafe4ab45f19bc073373c5eed318a7" exitCode=0 Nov 03 22:26:37.048256 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:37.047874 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" event={"ID":"a7974b57-e611-42b8-b320-3c82f1146d06","Type":"ContainerDied","Data":"f1d401f75979a8b7352e0be203549f18ecaafe4ab45f19bc073373c5eed318a7"} Nov 03 22:26:38.054353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:38.054317 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" event={"ID":"a7974b57-e611-42b8-b320-3c82f1146d06","Type":"ContainerStarted","Data":"e70ea8e2478ec583bed81d69e19e3a6c0349b41325300e803913893b94983004"} Nov 03 22:26:38.054731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:38.054403 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:38.197685 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:38.197588 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" podStartSLOduration=2.158673162 podStartE2EDuration="7.197573026s" totalImagesPullingTime="4.468925389s" totalInitContainerRuntime="569.974475ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:38.196210071 +0000 UTC m=+241.408979139" watchObservedRunningTime="2026-11-03 22:26:38.197573026 +0000 UTC m=+241.410342059" Nov 03 22:26:49.060854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:49.060822 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-9gvrp" Nov 03 22:26:49.834013 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:26:49.833973 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:26:54.857406 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.857372 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj"] Nov 03 22:26:54.859620 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.859602 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:54.903593 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.903568 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnt9\" (UniqueName: \"kubernetes.io/projected/79d3678c-04b5-45dd-b582-14aa9db24f5d-kube-api-access-xnnt9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7grhj\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:54.903702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.903602 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d3678c-04b5-45dd-b582-14aa9db24f5d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7grhj\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:54.965783 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.965742 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:26:54.965783 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.965744 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lmmc9\"" Nov 03 22:26:54.966050 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.965920 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:26:54.967101 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:54.967081 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj"] Nov 03 22:26:55.004460 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:55.004418 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnt9\" (UniqueName: \"kubernetes.io/projected/79d3678c-04b5-45dd-b582-14aa9db24f5d-kube-api-access-xnnt9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7grhj\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:55.004460 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:55.004465 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d3678c-04b5-45dd-b582-14aa9db24f5d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7grhj\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:55.004827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:55.004782 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d3678c-04b5-45dd-b582-14aa9db24f5d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7grhj\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:55.066107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:55.066076 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnt9\" (UniqueName: \"kubernetes.io/projected/79d3678c-04b5-45dd-b582-14aa9db24f5d-kube-api-access-xnnt9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7grhj\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:55.171154 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:55.171056 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:26:55.648301 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:55.648276 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj"] Nov 03 22:26:55.650707 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:26:55.650675 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d3678c_04b5_45dd_b582_14aa9db24f5d.slice/crio-8938ad7e6ecb0418e78094e48913b8b4cb5f50a4497a85eaf6e188e803052b97 WatchSource:0}: Error finding container 8938ad7e6ecb0418e78094e48913b8b4cb5f50a4497a85eaf6e188e803052b97: Status 404 returned error can't find the container with id 8938ad7e6ecb0418e78094e48913b8b4cb5f50a4497a85eaf6e188e803052b97 Nov 03 22:26:56.131341 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:56.131307 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" event={"ID":"79d3678c-04b5-45dd-b582-14aa9db24f5d","Type":"ContainerStarted","Data":"8938ad7e6ecb0418e78094e48913b8b4cb5f50a4497a85eaf6e188e803052b97"} Nov 03 22:26:58.870837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:58.870775 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-272st"] Nov 03 22:26:58.877610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:58.877562 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:26:58.943989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:58.943946 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkh6\" (UniqueName: \"kubernetes.io/projected/3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e-kube-api-access-ngkh6\") pod \"authorino-operator-7b7bb4bc77-272st\" (UID: \"3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:26:58.986531 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:58.986488 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-272st"] Nov 03 22:26:59.028665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:59.028635 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-c2k94\"" Nov 03 22:26:59.044926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:59.044895 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkh6\" (UniqueName: \"kubernetes.io/projected/3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e-kube-api-access-ngkh6\") pod \"authorino-operator-7b7bb4bc77-272st\" (UID: \"3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:26:59.110130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:59.110099 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkh6\" (UniqueName: \"kubernetes.io/projected/3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e-kube-api-access-ngkh6\") pod \"authorino-operator-7b7bb4bc77-272st\" (UID: \"3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:26:59.194108 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:59.194027 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:26:59.581283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:26:59.581250 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-272st"] Nov 03 22:26:59.584504 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:26:59.584472 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec61a1c_adac_4fa3_ac3a_b2cb06f1058e.slice/crio-484dedd7d6f08c2b1af80d450bc91ee13c79d3daa0eaf39c3d9732362d3b9ade WatchSource:0}: Error finding container 484dedd7d6f08c2b1af80d450bc91ee13c79d3daa0eaf39c3d9732362d3b9ade: Status 404 returned error can't find the container with id 484dedd7d6f08c2b1af80d450bc91ee13c79d3daa0eaf39c3d9732362d3b9ade Nov 03 22:27:00.152511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:00.152472 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" event={"ID":"3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e","Type":"ContainerStarted","Data":"484dedd7d6f08c2b1af80d450bc91ee13c79d3daa0eaf39c3d9732362d3b9ade"} Nov 03 22:27:02.164030 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.163990 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" event={"ID":"79d3678c-04b5-45dd-b582-14aa9db24f5d","Type":"ContainerStarted","Data":"2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db"} Nov 03 22:27:02.164587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.164098 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:27:02.405211 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.405158 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" podStartSLOduration=2.813876841 podStartE2EDuration="8.405141312s" totalImagesPullingTime="5.591264471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:02.354289007 +0000 UTC m=+265.567058059" watchObservedRunningTime="2026-11-03 22:27:02.405141312 +0000 UTC m=+265.617910362" Nov 03 22:27:02.680512 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.680479 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4"] Nov 03 22:27:02.682612 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.682593 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" Nov 03 22:27:02.741608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.741575 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8rxc5\"" Nov 03 22:27:02.745750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.745721 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 22:27:02.747589 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.747561 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4"] Nov 03 22:27:02.780302 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.780278 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvbk\" (UniqueName: \"kubernetes.io/projected/8ae1270c-80be-4215-b91e-c5cb1d0462c1-kube-api-access-zbvbk\") pod \"dns-operator-controller-manager-8557cbdb84-czcw4\" (UID: \"8ae1270c-80be-4215-b91e-c5cb1d0462c1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" Nov 03 22:27:02.881744 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.881702 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvbk\" (UniqueName: \"kubernetes.io/projected/8ae1270c-80be-4215-b91e-c5cb1d0462c1-kube-api-access-zbvbk\") pod \"dns-operator-controller-manager-8557cbdb84-czcw4\" (UID: \"8ae1270c-80be-4215-b91e-c5cb1d0462c1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" Nov 03 22:27:02.946493 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.946425 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvbk\" (UniqueName: \"kubernetes.io/projected/8ae1270c-80be-4215-b91e-c5cb1d0462c1-kube-api-access-zbvbk\") pod \"dns-operator-controller-manager-8557cbdb84-czcw4\" (UID: \"8ae1270c-80be-4215-b91e-c5cb1d0462c1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" Nov 03 22:27:02.994722 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:02.994688 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" Nov 03 22:27:03.170937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:03.170898 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" event={"ID":"3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e","Type":"ContainerStarted","Data":"0ad5c8080ffddc6abdcaa993b066dc32507595c0949e8b06447b6bc24c7df572"} Nov 03 22:27:03.171424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:03.171143 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:27:03.300783 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:03.300686 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" podStartSLOduration=2.181837988 podStartE2EDuration="5.300673477s" totalImagesPullingTime="3.118835489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:03.300445507 +0000 UTC m=+266.513214577" watchObservedRunningTime="2026-11-03 22:27:03.300673477 +0000 UTC m=+266.513442528" Nov 03 22:27:03.456577 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:03.456551 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4"] Nov 03 22:27:03.459091 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:27:03.459061 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae1270c_80be_4215_b91e_c5cb1d0462c1.slice/crio-fea306b4531d262a6da3d003bae40b60904e53fa79dd26dc21082c3228585078 WatchSource:0}: Error finding container fea306b4531d262a6da3d003bae40b60904e53fa79dd26dc21082c3228585078: Status 404 returned error can't find the container with id fea306b4531d262a6da3d003bae40b60904e53fa79dd26dc21082c3228585078 Nov 03 22:27:04.177024 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:04.176989 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" event={"ID":"8ae1270c-80be-4215-b91e-c5cb1d0462c1","Type":"ContainerStarted","Data":"fea306b4531d262a6da3d003bae40b60904e53fa79dd26dc21082c3228585078"} Nov 03 22:27:05.718739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:05.718703 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw"] Nov 03 22:27:05.721317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:05.721295 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" Nov 03 22:27:05.806898 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:05.806759 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n44km\" (UniqueName: \"kubernetes.io/projected/3ee52919-e882-49c4-967c-3a60a0632a47-kube-api-access-n44km\") pod \"limitador-operator-controller-manager-5778bb4449-9w4hw\" (UID: \"3ee52919-e882-49c4-967c-3a60a0632a47\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" Nov 03 22:27:05.834656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:05.834613 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5hwxw\"" Nov 03 22:27:05.836695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:05.836663 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw"] Nov 03 22:27:05.907552 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:05.907511 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n44km\" (UniqueName: \"kubernetes.io/projected/3ee52919-e882-49c4-967c-3a60a0632a47-kube-api-access-n44km\") pod \"limitador-operator-controller-manager-5778bb4449-9w4hw\" (UID: \"3ee52919-e882-49c4-967c-3a60a0632a47\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" Nov 03 22:27:05.970824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:05.970739 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n44km\" (UniqueName: \"kubernetes.io/projected/3ee52919-e882-49c4-967c-3a60a0632a47-kube-api-access-n44km\") pod \"limitador-operator-controller-manager-5778bb4449-9w4hw\" (UID: \"3ee52919-e882-49c4-967c-3a60a0632a47\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" Nov 03 22:27:06.037015 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:06.036942 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" Nov 03 22:27:06.421186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:06.421160 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw"] Nov 03 22:27:06.423259 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:27:06.423230 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee52919_e882_49c4_967c_3a60a0632a47.slice/crio-1365b58599e527cc74b3bc7b5e0fbf34a8d80fbdd0de22460a8206427c0f38f9 WatchSource:0}: Error finding container 1365b58599e527cc74b3bc7b5e0fbf34a8d80fbdd0de22460a8206427c0f38f9: Status 404 returned error can't find the container with id 1365b58599e527cc74b3bc7b5e0fbf34a8d80fbdd0de22460a8206427c0f38f9 Nov 03 22:27:07.190597 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:07.190562 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" event={"ID":"3ee52919-e882-49c4-967c-3a60a0632a47","Type":"ContainerStarted","Data":"1365b58599e527cc74b3bc7b5e0fbf34a8d80fbdd0de22460a8206427c0f38f9"} Nov 03 22:27:08.195968 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:08.195922 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" event={"ID":"8ae1270c-80be-4215-b91e-c5cb1d0462c1","Type":"ContainerStarted","Data":"1d54f6d00f409026e8ca8234e83270aee2c53246876f4796e1ae168cd15bb63c"} Nov 03 22:27:08.196409 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:08.195992 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" Nov 03 22:27:08.314599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:08.314542 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" podStartSLOduration=2.474435651 podStartE2EDuration="6.314529011s" totalImagesPullingTime="3.84009336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:08.313910393 +0000 UTC m=+271.526679458" watchObservedRunningTime="2026-11-03 22:27:08.314529011 +0000 UTC m=+271.527298061" Nov 03 22:27:13.173184 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.173152 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:27:13.697162 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.697129 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx"] Nov 03 22:27:13.700833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.700813 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:13.775330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.775298 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbc9\" (UniqueName: \"kubernetes.io/projected/37b16595-3519-44cd-ae29-0e25b5e30edf-kube-api-access-pxbc9\") pod \"kuadrant-operator-controller-manager-678d87766f-cr4wx\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:13.775330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.775338 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37b16595-3519-44cd-ae29-0e25b5e30edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cr4wx\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:13.783892 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.783863 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx"] Nov 03 22:27:13.876777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.876739 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbc9\" (UniqueName: \"kubernetes.io/projected/37b16595-3519-44cd-ae29-0e25b5e30edf-kube-api-access-pxbc9\") pod \"kuadrant-operator-controller-manager-678d87766f-cr4wx\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:13.876777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.876781 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37b16595-3519-44cd-ae29-0e25b5e30edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cr4wx\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:13.877180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.877159 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37b16595-3519-44cd-ae29-0e25b5e30edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cr4wx\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:13.938140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:13.938108 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbc9\" (UniqueName: \"kubernetes.io/projected/37b16595-3519-44cd-ae29-0e25b5e30edf-kube-api-access-pxbc9\") pod \"kuadrant-operator-controller-manager-678d87766f-cr4wx\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:14.012742 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:14.012662 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:14.179229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:14.179199 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:27:14.442331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:14.442289 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx"] Nov 03 22:27:14.446828 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:27:14.446039 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b16595_3519_44cd_ae29_0e25b5e30edf.slice/crio-e7c22bf58cd5feddc1fb4ade5b2d0e8a9a115c661c75d8bbac57570fb6144800 WatchSource:0}: Error finding container e7c22bf58cd5feddc1fb4ade5b2d0e8a9a115c661c75d8bbac57570fb6144800: Status 404 returned error can't find the container with id e7c22bf58cd5feddc1fb4ade5b2d0e8a9a115c661c75d8bbac57570fb6144800 Nov 03 22:27:15.227781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:15.227744 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" event={"ID":"37b16595-3519-44cd-ae29-0e25b5e30edf","Type":"ContainerStarted","Data":"402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0"} Nov 03 22:27:15.228194 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:15.227822 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" event={"ID":"37b16595-3519-44cd-ae29-0e25b5e30edf","Type":"ContainerStarted","Data":"e7c22bf58cd5feddc1fb4ade5b2d0e8a9a115c661c75d8bbac57570fb6144800"} Nov 03 22:27:15.228194 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:15.227851 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:15.448974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:15.448924 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" podStartSLOduration=2.448908638 podStartE2EDuration="2.448908638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:15.397858152 +0000 UTC m=+278.610627216" watchObservedRunningTime="2026-11-03 22:27:15.448908638 +0000 UTC m=+278.661677689" Nov 03 22:27:19.202836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:19.202805 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-czcw4" Nov 03 22:27:19.247042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:19.247007 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" event={"ID":"3ee52919-e882-49c4-967c-3a60a0632a47","Type":"ContainerStarted","Data":"d8d0e03fe195fae6f9971019f109d4f36af898b6348bb4fac0f7732466f18c7b"} Nov 03 22:27:19.247219 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:19.247139 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" Nov 03 22:27:19.437296 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:19.437235 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" podStartSLOduration=2.067077523 podStartE2EDuration="14.437219403s" totalImagesPullingTime="12.37014188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:19.436964468 +0000 UTC m=+282.649733548" watchObservedRunningTime="2026-11-03 22:27:19.437219403 +0000 UTC m=+282.649988453" Nov 03 22:27:19.834324 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:27:19.834286 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:27:26.235026 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.234994 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 22:27:26.458185 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.458145 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj"] Nov 03 22:27:26.458401 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.458365 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" podUID="79d3678c-04b5-45dd-b582-14aa9db24f5d" containerName="manager" containerID="cri-o://2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db" gracePeriod=10 Nov 03 22:27:26.709142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.709115 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:27:26.895293 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.895222 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d3678c-04b5-45dd-b582-14aa9db24f5d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d3678c-04b5-45dd-b582-14aa9db24f5d-extensions-socket-volume\") pod \"79d3678c-04b5-45dd-b582-14aa9db24f5d\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " Nov 03 22:27:26.895293 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.895277 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d3678c-04b5-45dd-b582-14aa9db24f5d-kube-api-access-xnnt9\" (UniqueName: \"kubernetes.io/projected/79d3678c-04b5-45dd-b582-14aa9db24f5d-kube-api-access-xnnt9\") pod \"79d3678c-04b5-45dd-b582-14aa9db24f5d\" (UID: \"79d3678c-04b5-45dd-b582-14aa9db24f5d\") " Nov 03 22:27:26.895699 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.895672 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d3678c-04b5-45dd-b582-14aa9db24f5d-extensions-socket-volume" pod "79d3678c-04b5-45dd-b582-14aa9db24f5d" (UID: "79d3678c-04b5-45dd-b582-14aa9db24f5d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:26.897451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.897424 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d3678c-04b5-45dd-b582-14aa9db24f5d-kube-api-access-xnnt9" pod "79d3678c-04b5-45dd-b582-14aa9db24f5d" (UID: "79d3678c-04b5-45dd-b582-14aa9db24f5d"). InnerVolumeSpecName "kube-api-access-xnnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:26.996408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.996377 4898 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d3678c-04b5-45dd-b582-14aa9db24f5d-extensions-socket-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:26.996408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:26.996408 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnnt9\" (UniqueName: \"kubernetes.io/projected/79d3678c-04b5-45dd-b582-14aa9db24f5d-kube-api-access-xnnt9\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:27.284000 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.283915 4898 generic.go:356] "Generic (PLEG): container finished" podID="79d3678c-04b5-45dd-b582-14aa9db24f5d" containerID="2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db" exitCode=0 Nov 03 22:27:27.284000 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.283969 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" Nov 03 22:27:27.284374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.283965 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" event={"ID":"79d3678c-04b5-45dd-b582-14aa9db24f5d","Type":"ContainerDied","Data":"2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db"} Nov 03 22:27:27.284374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.284178 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj" event={"ID":"79d3678c-04b5-45dd-b582-14aa9db24f5d","Type":"ContainerDied","Data":"8938ad7e6ecb0418e78094e48913b8b4cb5f50a4497a85eaf6e188e803052b97"} Nov 03 22:27:27.284374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.284209 4898 scope.go:122] "RemoveContainer" containerID="2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db" Nov 03 22:27:27.292672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.292654 4898 scope.go:122] "RemoveContainer" containerID="2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db" Nov 03 22:27:27.292966 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:27:27.292948 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db\": container with ID starting with 2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db not found: ID does not exist" containerID="2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db" Nov 03 22:27:27.293012 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.292976 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db"} err="failed to get container status \"2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db\": rpc error: code = NotFound desc = could not find container \"2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db\": container with ID starting with 2b6d991f653b16933e4c05c0450ddc7e1d5e58201032b02e55549a7da0f421db not found: ID does not exist" Nov 03 22:27:27.461493 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.461456 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj"] Nov 03 22:27:27.511728 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:27.511692 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7grhj"] Nov 03 22:27:28.442617 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.442582 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6"] Nov 03 22:27:28.443031 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.442964 4898 cpu_manager.go:414] "containerMap: removing container" podUID="79d3678c-04b5-45dd-b582-14aa9db24f5d" containerName="manager" Nov 03 22:27:28.443031 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.442977 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d3678c-04b5-45dd-b582-14aa9db24f5d" containerName="manager" Nov 03 22:27:28.443103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.443072 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d3678c-04b5-45dd-b582-14aa9db24f5d" containerName="manager" Nov 03 22:27:28.447290 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.447270 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.500177 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.500141 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:27:28.500345 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.500200 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:27:28.550180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.550147 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-mbqbd\"" Nov 03 22:27:28.610908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.610876 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.610908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.610914 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.611111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.610987 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtwq\" (UniqueName: \"kubernetes.io/projected/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-kube-api-access-vmtwq\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.611111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.611050 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.651370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.651334 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 22:27:28.652979 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.652956 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6"] Nov 03 22:27:28.711918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.711852 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.711918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.711900 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.712060 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.712027 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.712104 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.712087 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtwq\" (UniqueName: \"kubernetes.io/projected/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-kube-api-access-vmtwq\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.712320 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.712299 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.712408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.712296 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.713011 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.712995 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.773223 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.773194 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtwq\" (UniqueName: \"kubernetes.io/projected/cee24d57-c7bb-4335-8a5e-9cd5ef61ae26-kube-api-access-vmtwq\") pod \"kserve-module-controller-manager-7566cf5fc8-zkkb6\" (UID: \"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:28.904971 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:28.904936 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d3678c-04b5-45dd-b582-14aa9db24f5d" path="/var/lib/kubelet/pods/79d3678c-04b5-45dd-b582-14aa9db24f5d/volumes" Nov 03 22:27:29.059152 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:29.059083 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:29.503380 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:29.503355 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6"] Nov 03 22:27:29.505744 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:27:29.505718 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee24d57_c7bb_4335_8a5e_9cd5ef61ae26.slice/crio-e11ca4f9c180265629dbe29a93e496157cb706dea74d34ba39699495299b4c70 WatchSource:0}: Error finding container e11ca4f9c180265629dbe29a93e496157cb706dea74d34ba39699495299b4c70: Status 404 returned error can't find the container with id e11ca4f9c180265629dbe29a93e496157cb706dea74d34ba39699495299b4c70 Nov 03 22:27:30.253646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:30.253616 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9w4hw" Nov 03 22:27:30.299744 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:30.299704 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" event={"ID":"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26","Type":"ContainerStarted","Data":"e11ca4f9c180265629dbe29a93e496157cb706dea74d34ba39699495299b4c70"} Nov 03 22:27:33.317320 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:33.317284 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" event={"ID":"cee24d57-c7bb-4335-8a5e-9cd5ef61ae26","Type":"ContainerStarted","Data":"4a67f0e2dca76877d87ce2e3a3a2f24e0dd6fdaa2b391abf4bc19e0e06eec485"} Nov 03 22:27:33.317834 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:33.317436 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:27:33.489286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:33.489218 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" podStartSLOduration=2.48652607 podStartE2EDuration="5.489197775s" totalImagesPullingTime="3.002671705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:33.486875148 +0000 UTC m=+296.699644195" watchObservedRunningTime="2026-11-03 22:27:33.489197775 +0000 UTC m=+296.701966831" Nov 03 22:27:36.926951 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:36.926926 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:27:36.927301 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:36.927068 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:27:36.931560 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:36.931541 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:27:36.931947 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:36.931922 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:27:45.886749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:45.886717 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pssqj"] Nov 03 22:27:45.894306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:45.894275 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:45.945884 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:45.945855 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pssqj"] Nov 03 22:27:45.960507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:45.960475 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbqx\" (UniqueName: \"kubernetes.io/projected/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-kube-api-access-slbqx\") pod \"limitador-limitador-74dfcfb894-pssqj\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:45.960646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:45.960586 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-config-file\") pod \"limitador-limitador-74dfcfb894-pssqj\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:45.994677 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:45.994655 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 22:27:46.022527 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.022499 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5plgf\"" Nov 03 22:27:46.061340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.061312 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbqx\" (UniqueName: \"kubernetes.io/projected/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-kube-api-access-slbqx\") pod \"limitador-limitador-74dfcfb894-pssqj\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:46.061463 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.061357 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-config-file\") pod \"limitador-limitador-74dfcfb894-pssqj\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:46.061952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.061935 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-config-file\") pod \"limitador-limitador-74dfcfb894-pssqj\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:46.134524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.134492 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pssqj"] Nov 03 22:27:46.137413 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.137355 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbqx\" (UniqueName: \"kubernetes.io/projected/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-kube-api-access-slbqx\") pod \"limitador-limitador-74dfcfb894-pssqj\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:46.206179 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.206143 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:46.579116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.579092 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pssqj"] Nov 03 22:27:46.581395 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:27:46.581367 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8f17b3_bbc3_4de3_95a3_839f31963f5a.slice/crio-8f0e1bcdac8e7789452798f62ed44efd07ff4855f5060613fb6eef84df840ec8 WatchSource:0}: Error finding container 8f0e1bcdac8e7789452798f62ed44efd07ff4855f5060613fb6eef84df840ec8: Status 404 returned error can't find the container with id 8f0e1bcdac8e7789452798f62ed44efd07ff4855f5060613fb6eef84df840ec8 Nov 03 22:27:46.583236 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:46.583219 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:27:47.380929 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:47.380890 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" event={"ID":"5c8f17b3-bbc3-4de3-95a3-839f31963f5a","Type":"ContainerStarted","Data":"8f0e1bcdac8e7789452798f62ed44efd07ff4855f5060613fb6eef84df840ec8"} Nov 03 22:27:49.391965 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:49.391925 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" event={"ID":"5c8f17b3-bbc3-4de3-95a3-839f31963f5a","Type":"ContainerStarted","Data":"752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f"} Nov 03 22:27:49.392381 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:49.392002 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:27:49.512330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:49.512280 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" podStartSLOduration=2.242932652 podStartE2EDuration="4.512263478s" totalImagesPullingTime="2.269330826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:49.510532278 +0000 UTC m=+312.723301328" watchObservedRunningTime="2026-11-03 22:27:49.512263478 +0000 UTC m=+312.725032529" Nov 03 22:27:49.834688 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:27:49.834648 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:27:54.322935 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:27:54.322906 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zkkb6" Nov 03 22:28:00.398163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:00.398124 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:28:04.580813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:04.580745 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pssqj"] Nov 03 22:28:04.581243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:04.580982 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" podUID="5c8f17b3-bbc3-4de3-95a3-839f31963f5a" containerName="limitador" containerID="cri-o://752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f" gracePeriod=30 Nov 03 22:28:05.128766 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.128739 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:28:05.229976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.229897 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-kube-api-access-slbqx\" (UniqueName: \"kubernetes.io/projected/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-kube-api-access-slbqx\") pod \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " Nov 03 22:28:05.229976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.229946 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-config-file\" (UniqueName: \"kubernetes.io/configmap/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-config-file\") pod \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\" (UID: \"5c8f17b3-bbc3-4de3-95a3-839f31963f5a\") " Nov 03 22:28:05.230312 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.230287 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-config-file" pod "5c8f17b3-bbc3-4de3-95a3-839f31963f5a" (UID: "5c8f17b3-bbc3-4de3-95a3-839f31963f5a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:05.232029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.232006 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-kube-api-access-slbqx" pod "5c8f17b3-bbc3-4de3-95a3-839f31963f5a" (UID: "5c8f17b3-bbc3-4de3-95a3-839f31963f5a"). InnerVolumeSpecName "kube-api-access-slbqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:05.330580 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.330547 4898 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-config-file\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:05.330580 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.330579 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbqx\" (UniqueName: \"kubernetes.io/projected/5c8f17b3-bbc3-4de3-95a3-839f31963f5a-kube-api-access-slbqx\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:05.467636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.467589 4898 generic.go:356] "Generic (PLEG): container finished" podID="5c8f17b3-bbc3-4de3-95a3-839f31963f5a" containerID="752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f" exitCode=0 Nov 03 22:28:05.467636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.467636 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" Nov 03 22:28:05.467882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.467667 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" event={"ID":"5c8f17b3-bbc3-4de3-95a3-839f31963f5a","Type":"ContainerDied","Data":"752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f"} Nov 03 22:28:05.467882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.467724 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pssqj" event={"ID":"5c8f17b3-bbc3-4de3-95a3-839f31963f5a","Type":"ContainerDied","Data":"8f0e1bcdac8e7789452798f62ed44efd07ff4855f5060613fb6eef84df840ec8"} Nov 03 22:28:05.467882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.467747 4898 scope.go:122] "RemoveContainer" containerID="752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f" Nov 03 22:28:05.476511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.476487 4898 scope.go:122] "RemoveContainer" containerID="752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f" Nov 03 22:28:05.476812 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:28:05.476775 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f\": container with ID starting with 752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f not found: ID does not exist" containerID="752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f" Nov 03 22:28:05.476891 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.476822 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f"} err="failed to get container status \"752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f\": rpc error: code = NotFound desc = could not find container \"752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f\": container with ID starting with 752660425e927499581ad1e50acdd37f73e02dd1ca3c3dffaec682a771c3793f not found: ID does not exist" Nov 03 22:28:05.633538 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.633500 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pssqj"] Nov 03 22:28:05.636745 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:05.636714 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pssqj"] Nov 03 22:28:06.904991 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:06.904953 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8f17b3-bbc3-4de3-95a3-839f31963f5a" path="/var/lib/kubelet/pods/5c8f17b3-bbc3-4de3-95a3-839f31963f5a/volumes" Nov 03 22:28:10.940775 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:10.940731 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc"] Nov 03 22:28:10.941312 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:10.941292 4898 cpu_manager.go:414] "containerMap: removing container" podUID="5c8f17b3-bbc3-4de3-95a3-839f31963f5a" containerName="limitador" Nov 03 22:28:10.941366 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:10.941316 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8f17b3-bbc3-4de3-95a3-839f31963f5a" containerName="limitador" Nov 03 22:28:10.941441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:10.941428 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8f17b3-bbc3-4de3-95a3-839f31963f5a" containerName="limitador" Nov 03 22:28:10.944417 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:10.944388 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.000548 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.000510 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc"] Nov 03 22:28:11.003710 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.003687 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:11.048608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.048578 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 22:28:11.049642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.049627 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:11.049760 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.049644 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gvnxw\"" Nov 03 22:28:11.079884 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.079855 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxqg\" (UniqueName: \"kubernetes.io/projected/a0861fcf-b07c-4b37-a7be-05855af26583-kube-api-access-lxxqg\") pod \"postgres-7cdbf6dc44-qsvkc\" (UID: \"a0861fcf-b07c-4b37-a7be-05855af26583\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.080056 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.079997 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0861fcf-b07c-4b37-a7be-05855af26583-data\") pod \"postgres-7cdbf6dc44-qsvkc\" (UID: \"a0861fcf-b07c-4b37-a7be-05855af26583\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.181375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.181339 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxqg\" (UniqueName: \"kubernetes.io/projected/a0861fcf-b07c-4b37-a7be-05855af26583-kube-api-access-lxxqg\") pod \"postgres-7cdbf6dc44-qsvkc\" (UID: \"a0861fcf-b07c-4b37-a7be-05855af26583\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.181538 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.181444 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0861fcf-b07c-4b37-a7be-05855af26583-data\") pod \"postgres-7cdbf6dc44-qsvkc\" (UID: \"a0861fcf-b07c-4b37-a7be-05855af26583\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.181865 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.181848 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0861fcf-b07c-4b37-a7be-05855af26583-data\") pod \"postgres-7cdbf6dc44-qsvkc\" (UID: \"a0861fcf-b07c-4b37-a7be-05855af26583\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.251737 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.251671 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxqg\" (UniqueName: \"kubernetes.io/projected/a0861fcf-b07c-4b37-a7be-05855af26583-kube-api-access-lxxqg\") pod \"postgres-7cdbf6dc44-qsvkc\" (UID: \"a0861fcf-b07c-4b37-a7be-05855af26583\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.258119 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.258096 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:11.630329 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:11.630306 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc"] Nov 03 22:28:11.632430 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:28:11.632404 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0861fcf_b07c_4b37_a7be_05855af26583.slice/crio-66d5d2df42dd3fa650114ea6cc28809837055ebb07893115b1ec98dbf02f2db0 WatchSource:0}: Error finding container 66d5d2df42dd3fa650114ea6cc28809837055ebb07893115b1ec98dbf02f2db0: Status 404 returned error can't find the container with id 66d5d2df42dd3fa650114ea6cc28809837055ebb07893115b1ec98dbf02f2db0 Nov 03 22:28:12.502006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:12.501973 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" event={"ID":"a0861fcf-b07c-4b37-a7be-05855af26583","Type":"ContainerStarted","Data":"66d5d2df42dd3fa650114ea6cc28809837055ebb07893115b1ec98dbf02f2db0"} Nov 03 22:28:19.538146 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:19.538103 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" event={"ID":"a0861fcf-b07c-4b37-a7be-05855af26583","Type":"ContainerStarted","Data":"a8cba0a6ed312d27ddc6ac58ba2066f15130d01d3b205a92213ce4d1dce93b62"} Nov 03 22:28:19.538715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:19.538303 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:19.657969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:19.657916 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" podStartSLOduration=2.405751437 podStartE2EDuration="9.657902167s" totalImagesPullingTime="7.25215073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:19.656555847 +0000 UTC m=+342.869324899" watchObservedRunningTime="2026-11-03 22:28:19.657902167 +0000 UTC m=+342.870671218" Nov 03 22:28:19.834912 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:28:19.834879 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:28:25.572043 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:25.572013 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qsvkc" Nov 03 22:28:32.886974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.886938 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gjbq8"] Nov 03 22:28:32.894209 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.894180 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:32.951543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.951509 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gjbq8"] Nov 03 22:28:32.972841 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.972801 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvg78\" (UniqueName: \"kubernetes.io/projected/2844371b-1ae7-44ac-bf64-553356402921-kube-api-access-zvg78\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:32.973001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.972933 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-metrics-certs\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:32.973001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.972994 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-webhook-cert\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:32.999340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.999307 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 22:28:32.999498 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.999412 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6fd48\"" Nov 03 22:28:32.999498 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:32.999434 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:28:33.000513 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.000492 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 22:28:33.074166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.074131 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-webhook-cert\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:33.074347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.074190 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvg78\" (UniqueName: \"kubernetes.io/projected/2844371b-1ae7-44ac-bf64-553356402921-kube-api-access-zvg78\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:33.074347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.074254 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-metrics-certs\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:33.076772 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.076746 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-webhook-cert\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:33.076772 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.076758 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-metrics-certs\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:33.135783 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.135748 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvg78\" (UniqueName: \"kubernetes.io/projected/2844371b-1ae7-44ac-bf64-553356402921-kube-api-access-zvg78\") pod \"maas-controller-579cf7c6d-gjbq8\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:33.207278 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.207192 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:33.331539 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.331503 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b656c5c54-p66vm"] Nov 03 22:28:33.336199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.336180 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.376520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.376490 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pss\" (UniqueName: \"kubernetes.io/projected/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-kube-api-access-f9pss\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.376648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.376606 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-metrics-certs\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.376648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.376633 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-webhook-cert\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.392439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.392411 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b656c5c54-p66vm"] Nov 03 22:28:33.477958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.477877 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-metrics-certs\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.477958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.477931 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-webhook-cert\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.478181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.477990 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pss\" (UniqueName: \"kubernetes.io/projected/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-kube-api-access-f9pss\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.480390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.480365 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-metrics-certs\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.480555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.480537 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-webhook-cert\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.539562 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.539527 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pss\" (UniqueName: \"kubernetes.io/projected/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-kube-api-access-f9pss\") pod \"maas-controller-b656c5c54-p66vm\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:33.580951 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.580926 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gjbq8"] Nov 03 22:28:33.582914 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:28:33.582888 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2844371b_1ae7_44ac_bf64_553356402921.slice/crio-ebbbccc520fff07e796d91f5ee2815c4b410ccb91caecbd36c8b9aa60899eadf WatchSource:0}: Error finding container ebbbccc520fff07e796d91f5ee2815c4b410ccb91caecbd36c8b9aa60899eadf: Status 404 returned error can't find the container with id ebbbccc520fff07e796d91f5ee2815c4b410ccb91caecbd36c8b9aa60899eadf Nov 03 22:28:33.601720 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.601690 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" event={"ID":"2844371b-1ae7-44ac-bf64-553356402921","Type":"ContainerStarted","Data":"ebbbccc520fff07e796d91f5ee2815c4b410ccb91caecbd36c8b9aa60899eadf"} Nov 03 22:28:33.649037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:33.649015 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:34.025317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:34.025291 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b656c5c54-p66vm"] Nov 03 22:28:34.027390 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:28:34.027362 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ff4bb6_8c87_4027_baf5_0a80a0b44261.slice/crio-00cd435ed2b03af9df2cb1c2fb5e68ce7735cee3382bf53ae953ba4aeee3bddb WatchSource:0}: Error finding container 00cd435ed2b03af9df2cb1c2fb5e68ce7735cee3382bf53ae953ba4aeee3bddb: Status 404 returned error can't find the container with id 00cd435ed2b03af9df2cb1c2fb5e68ce7735cee3382bf53ae953ba4aeee3bddb Nov 03 22:28:34.611988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:34.611948 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-p66vm" event={"ID":"f7ff4bb6-8c87-4027-baf5-0a80a0b44261","Type":"ContainerStarted","Data":"00cd435ed2b03af9df2cb1c2fb5e68ce7735cee3382bf53ae953ba4aeee3bddb"} Nov 03 22:28:36.911259 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:36.911228 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xnpgs"] Nov 03 22:28:36.918857 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:36.918834 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:36.970694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:36.970658 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xnpgs"] Nov 03 22:28:37.016536 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.016498 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97e3746c-a9f4-42f2-8dc4-2b62340078df-cert\") pod \"kserve-controller-manager-776cc66d47-xnpgs\" (UID: \"97e3746c-a9f4-42f2-8dc4-2b62340078df\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:37.016717 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.016664 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzshw\" (UniqueName: \"kubernetes.io/projected/97e3746c-a9f4-42f2-8dc4-2b62340078df-kube-api-access-zzshw\") pod \"kserve-controller-manager-776cc66d47-xnpgs\" (UID: \"97e3746c-a9f4-42f2-8dc4-2b62340078df\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:37.024819 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.018776 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 22:28:37.024819 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.018871 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vnm5r\"" Nov 03 22:28:37.028008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.027981 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq"] Nov 03 22:28:37.038183 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.038158 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.088569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.088530 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq"] Nov 03 22:28:37.091127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.091099 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-66c2f\"" Nov 03 22:28:37.091268 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.091147 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 22:28:37.118174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.118130 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f6791d-a54e-4662-91f7-c1edf645f7e4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r6znq\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.118317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.118204 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzshw\" (UniqueName: \"kubernetes.io/projected/97e3746c-a9f4-42f2-8dc4-2b62340078df-kube-api-access-zzshw\") pod \"kserve-controller-manager-776cc66d47-xnpgs\" (UID: \"97e3746c-a9f4-42f2-8dc4-2b62340078df\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:37.118317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.118268 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqps\" (UniqueName: \"kubernetes.io/projected/91f6791d-a54e-4662-91f7-c1edf645f7e4-kube-api-access-zqqps\") pod \"llmisvc-controller-manager-869d5dcc5d-r6znq\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.118411 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.118343 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97e3746c-a9f4-42f2-8dc4-2b62340078df-cert\") pod \"kserve-controller-manager-776cc66d47-xnpgs\" (UID: \"97e3746c-a9f4-42f2-8dc4-2b62340078df\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:37.121743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.121716 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97e3746c-a9f4-42f2-8dc4-2b62340078df-cert\") pod \"kserve-controller-manager-776cc66d47-xnpgs\" (UID: \"97e3746c-a9f4-42f2-8dc4-2b62340078df\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:37.179924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.179847 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzshw\" (UniqueName: \"kubernetes.io/projected/97e3746c-a9f4-42f2-8dc4-2b62340078df-kube-api-access-zzshw\") pod \"kserve-controller-manager-776cc66d47-xnpgs\" (UID: \"97e3746c-a9f4-42f2-8dc4-2b62340078df\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:37.219426 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.219385 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f6791d-a54e-4662-91f7-c1edf645f7e4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r6znq\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.219601 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.219459 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqps\" (UniqueName: \"kubernetes.io/projected/91f6791d-a54e-4662-91f7-c1edf645f7e4-kube-api-access-zqqps\") pod \"llmisvc-controller-manager-869d5dcc5d-r6znq\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.222894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.222863 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f6791d-a54e-4662-91f7-c1edf645f7e4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r6znq\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.236643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.236610 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:37.270606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.270574 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-bxcld"] Nov 03 22:28:37.275443 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.275421 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.284573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.284548 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqps\" (UniqueName: \"kubernetes.io/projected/91f6791d-a54e-4662-91f7-c1edf645f7e4-kube-api-access-zqqps\") pod \"llmisvc-controller-manager-869d5dcc5d-r6znq\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.321441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.321363 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a64a96-32b4-4535-b038-6610314c54e5-metrics-certs\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.321579 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.321516 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h767w\" (UniqueName: \"kubernetes.io/projected/a3a64a96-32b4-4535-b038-6610314c54e5-kube-api-access-h767w\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.321579 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.321557 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3a64a96-32b4-4535-b038-6610314c54e5-cert\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.329674 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.329647 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-88n5c\"" Nov 03 22:28:37.330913 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.330887 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 22:28:37.331055 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.331037 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 22:28:37.334484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.334459 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bxcld"] Nov 03 22:28:37.355130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.355104 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:37.424093 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.422391 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a64a96-32b4-4535-b038-6610314c54e5-metrics-certs\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.424093 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.422640 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h767w\" (UniqueName: \"kubernetes.io/projected/a3a64a96-32b4-4535-b038-6610314c54e5-kube-api-access-h767w\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.424093 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.422678 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3a64a96-32b4-4535-b038-6610314c54e5-cert\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.425744 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.425719 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3a64a96-32b4-4535-b038-6610314c54e5-cert\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.427221 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.427202 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a64a96-32b4-4535-b038-6610314c54e5-metrics-certs\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.486516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.486413 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h767w\" (UniqueName: \"kubernetes.io/projected/a3a64a96-32b4-4535-b038-6610314c54e5-kube-api-access-h767w\") pod \"odh-model-controller-769875db56-bxcld\" (UID: \"a3a64a96-32b4-4535-b038-6610314c54e5\") " pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.540140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.540113 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:28:37.595081 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.595056 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:37.627677 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.627587 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xnpgs"] Nov 03 22:28:37.632281 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:28:37.632253 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e3746c_a9f4_42f2_8dc4_2b62340078df.slice/crio-a7493d7fc9d51502942c010322c5990da75b87f159e0026eab681ab5ea8f4eca WatchSource:0}: Error finding container a7493d7fc9d51502942c010322c5990da75b87f159e0026eab681ab5ea8f4eca: Status 404 returned error can't find the container with id a7493d7fc9d51502942c010322c5990da75b87f159e0026eab681ab5ea8f4eca Nov 03 22:28:37.634746 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.634717 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" event={"ID":"2844371b-1ae7-44ac-bf64-553356402921","Type":"ContainerStarted","Data":"f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907"} Nov 03 22:28:37.634880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.634842 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:37.735841 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.735815 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq"] Nov 03 22:28:37.737513 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:28:37.737454 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod91f6791d_a54e_4662_91f7_c1edf645f7e4.slice/crio-73cb32c7720404fdd60991cc7fb508cf4a19bdd9da53849ddef127860029aeda WatchSource:0}: Error finding container 73cb32c7720404fdd60991cc7fb508cf4a19bdd9da53849ddef127860029aeda: Status 404 returned error can't find the container with id 73cb32c7720404fdd60991cc7fb508cf4a19bdd9da53849ddef127860029aeda Nov 03 22:28:37.754428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.754380 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" podStartSLOduration=1.8509962309999999 podStartE2EDuration="5.754358756s" totalImagesPullingTime="3.903362525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:37.752520045 +0000 UTC m=+360.965289096" watchObservedRunningTime="2026-11-03 22:28:37.754358756 +0000 UTC m=+360.967127806" Nov 03 22:28:37.980370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:37.980342 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bxcld"] Nov 03 22:28:37.982813 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:28:37.982770 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a64a96_32b4_4535_b038_6610314c54e5.slice/crio-23fa3803017812f500f3f58ce01ad6415fcd3b9e12fc728ed449b7aca4ee458e WatchSource:0}: Error finding container 23fa3803017812f500f3f58ce01ad6415fcd3b9e12fc728ed449b7aca4ee458e: Status 404 returned error can't find the container with id 23fa3803017812f500f3f58ce01ad6415fcd3b9e12fc728ed449b7aca4ee458e Nov 03 22:28:38.640873 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:38.640829 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" event={"ID":"97e3746c-a9f4-42f2-8dc4-2b62340078df","Type":"ContainerStarted","Data":"a7493d7fc9d51502942c010322c5990da75b87f159e0026eab681ab5ea8f4eca"} Nov 03 22:28:38.642018 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:38.641987 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bxcld" event={"ID":"a3a64a96-32b4-4535-b038-6610314c54e5","Type":"ContainerStarted","Data":"23fa3803017812f500f3f58ce01ad6415fcd3b9e12fc728ed449b7aca4ee458e"} Nov 03 22:28:38.643465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:38.643443 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-p66vm" event={"ID":"f7ff4bb6-8c87-4027-baf5-0a80a0b44261","Type":"ContainerStarted","Data":"0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6"} Nov 03 22:28:38.643622 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:38.643585 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:38.644711 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:38.644692 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" event={"ID":"91f6791d-a54e-4662-91f7-c1edf645f7e4","Type":"ContainerStarted","Data":"73cb32c7720404fdd60991cc7fb508cf4a19bdd9da53849ddef127860029aeda"} Nov 03 22:28:38.763029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:38.762955 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b656c5c54-p66vm" podStartSLOduration=2.159779322 podStartE2EDuration="5.762937425s" totalImagesPullingTime="3.603158103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:38.762269665 +0000 UTC m=+361.975038739" watchObservedRunningTime="2026-11-03 22:28:38.762937425 +0000 UTC m=+361.975706477" Nov 03 22:28:41.663216 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:41.663178 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" event={"ID":"91f6791d-a54e-4662-91f7-c1edf645f7e4","Type":"ContainerStarted","Data":"1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2"} Nov 03 22:28:41.663692 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:41.663379 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:28:41.664780 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:41.664755 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" event={"ID":"97e3746c-a9f4-42f2-8dc4-2b62340078df","Type":"ContainerStarted","Data":"1c5fec909f01b380ff372218e16bcdfedc06be0f7d6c451fdf0f3c13c6266e21"} Nov 03 22:28:41.664918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:41.664904 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:28:41.781546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:41.781481 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" podStartSLOduration=2.796308805 podStartE2EDuration="5.781461731s" totalImagesPullingTime="2.985152926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:41.779360556 +0000 UTC m=+364.992129628" watchObservedRunningTime="2026-11-03 22:28:41.781461731 +0000 UTC m=+364.994230783" Nov 03 22:28:41.896580 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:41.896531 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" podStartSLOduration=2.753661566 podStartE2EDuration="5.896517222s" totalImagesPullingTime="3.142855656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:41.895614376 +0000 UTC m=+365.108383459" watchObservedRunningTime="2026-11-03 22:28:41.896517222 +0000 UTC m=+365.109286274" Nov 03 22:28:42.670507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:42.670465 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bxcld" event={"ID":"a3a64a96-32b4-4535-b038-6610314c54e5","Type":"ContainerStarted","Data":"47eddad82bdd68a16dbeecf66d8529955439e5cbfcd16ce25d3cc481f9d7c555"} Nov 03 22:28:42.670938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:42.670891 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:28:42.788857 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:42.788806 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-bxcld" podStartSLOduration=2.099021264 podStartE2EDuration="5.788769828s" totalImagesPullingTime="3.689748564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:42.787747906 +0000 UTC m=+366.000516957" watchObservedRunningTime="2026-11-03 22:28:42.788769828 +0000 UTC m=+366.001538922" Nov 03 22:28:48.647090 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:48.647056 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:49.652924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:49.652891 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:28:49.824807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:49.824763 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gjbq8"] Nov 03 22:28:49.825048 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:49.825018 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" podUID="2844371b-1ae7-44ac-bf64-553356402921" containerName="manager" containerID="cri-o://f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907" gracePeriod=30 Nov 03 22:28:49.841691 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:28:49.841658 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:28:50.126593 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.126563 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:50.252516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.252414 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2844371b-1ae7-44ac-bf64-553356402921-kube-api-access-zvg78\" (UniqueName: \"kubernetes.io/projected/2844371b-1ae7-44ac-bf64-553356402921-kube-api-access-zvg78\") pod \"2844371b-1ae7-44ac-bf64-553356402921\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " Nov 03 22:28:50.252516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.252467 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-webhook-cert\") pod \"2844371b-1ae7-44ac-bf64-553356402921\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " Nov 03 22:28:50.252749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.252552 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-metrics-certs\") pod \"2844371b-1ae7-44ac-bf64-553356402921\" (UID: \"2844371b-1ae7-44ac-bf64-553356402921\") " Nov 03 22:28:50.254760 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.254730 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-metrics-certs" pod "2844371b-1ae7-44ac-bf64-553356402921" (UID: "2844371b-1ae7-44ac-bf64-553356402921"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:50.254941 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.254837 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-webhook-cert" pod "2844371b-1ae7-44ac-bf64-553356402921" (UID: "2844371b-1ae7-44ac-bf64-553356402921"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:50.254941 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.254831 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2844371b-1ae7-44ac-bf64-553356402921-kube-api-access-zvg78" pod "2844371b-1ae7-44ac-bf64-553356402921" (UID: "2844371b-1ae7-44ac-bf64-553356402921"). InnerVolumeSpecName "kube-api-access-zvg78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:50.354093 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.354052 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:50.354093 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.354090 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvg78\" (UniqueName: \"kubernetes.io/projected/2844371b-1ae7-44ac-bf64-553356402921-kube-api-access-zvg78\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:50.354306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.354108 4898 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2844371b-1ae7-44ac-bf64-553356402921-webhook-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:50.712123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.712085 4898 generic.go:356] "Generic (PLEG): container finished" podID="2844371b-1ae7-44ac-bf64-553356402921" containerID="f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907" exitCode=0 Nov 03 22:28:50.712608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.712132 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" event={"ID":"2844371b-1ae7-44ac-bf64-553356402921","Type":"ContainerDied","Data":"f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907"} Nov 03 22:28:50.712608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.712154 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" Nov 03 22:28:50.712608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.712222 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gjbq8" event={"ID":"2844371b-1ae7-44ac-bf64-553356402921","Type":"ContainerDied","Data":"ebbbccc520fff07e796d91f5ee2815c4b410ccb91caecbd36c8b9aa60899eadf"} Nov 03 22:28:50.712608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.712255 4898 scope.go:122] "RemoveContainer" containerID="f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907" Nov 03 22:28:50.722123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.722104 4898 scope.go:122] "RemoveContainer" containerID="f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907" Nov 03 22:28:50.722371 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:28:50.722352 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907\": container with ID starting with f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907 not found: ID does not exist" containerID="f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907" Nov 03 22:28:50.722423 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.722380 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907"} err="failed to get container status \"f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907\": rpc error: code = NotFound desc = could not find container \"f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907\": container with ID starting with f5673911b120667f294d2bea356cc891577f9996027625b07003d1a5421ec907 not found: ID does not exist" Nov 03 22:28:50.877187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.877158 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gjbq8"] Nov 03 22:28:50.927622 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:50.927592 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gjbq8"] Nov 03 22:28:52.905252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:52.905220 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2844371b-1ae7-44ac-bf64-553356402921" path="/var/lib/kubelet/pods/2844371b-1ae7-44ac-bf64-553356402921/volumes" Nov 03 22:28:54.684402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:28:54.684374 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-bxcld" Nov 03 22:29:12.672451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:12.672409 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:29:12.675394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:12.675368 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xnpgs" Nov 03 22:29:14.890903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:14.890866 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv"] Nov 03 22:29:14.891330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:14.891316 4898 cpu_manager.go:414] "containerMap: removing container" podUID="2844371b-1ae7-44ac-bf64-553356402921" containerName="manager" Nov 03 22:29:14.891372 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:14.891332 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2844371b-1ae7-44ac-bf64-553356402921" containerName="manager" Nov 03 22:29:14.891429 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:14.891420 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2844371b-1ae7-44ac-bf64-553356402921" containerName="manager" Nov 03 22:29:14.997698 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:14.997662 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv"] Nov 03 22:29:14.997903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:14.997724 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.072568 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.072534 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-maas-api-tls\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.072741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.072593 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611ea43-1944-4e72-aa7e-966a23812c7f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.072853 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.072832 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjdd\" (UniqueName: \"kubernetes.io/projected/3611ea43-1944-4e72-aa7e-966a23812c7f-kube-api-access-pbjdd\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.072925 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.072876 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-metrics-certs\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.105377 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.105350 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:29:15.105514 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.105382 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9qhjs\"" Nov 03 22:29:15.105514 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.105452 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 22:29:15.174149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.174080 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-maas-api-tls\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.174149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.174119 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611ea43-1944-4e72-aa7e-966a23812c7f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.174333 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.174312 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjdd\" (UniqueName: \"kubernetes.io/projected/3611ea43-1944-4e72-aa7e-966a23812c7f-kube-api-access-pbjdd\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.174396 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.174383 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-metrics-certs\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.174912 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.174889 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611ea43-1944-4e72-aa7e-966a23812c7f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.176648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.176624 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-metrics-certs\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.176727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.176670 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-maas-api-tls\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.235750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.235717 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjdd\" (UniqueName: \"kubernetes.io/projected/3611ea43-1944-4e72-aa7e-966a23812c7f-kube-api-access-pbjdd\") pod \"maas-api-6d669c7fcd-6nbfv\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.327119 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.327088 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:15.746631 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.746603 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv"] Nov 03 22:29:15.748446 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:29:15.748415 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3611ea43_1944_4e72_aa7e_966a23812c7f.slice/crio-a26a9afd55c907b5dd0f3f943a66cae1b6a44b02c37daace98f33bb0567b59d1 WatchSource:0}: Error finding container a26a9afd55c907b5dd0f3f943a66cae1b6a44b02c37daace98f33bb0567b59d1: Status 404 returned error can't find the container with id a26a9afd55c907b5dd0f3f943a66cae1b6a44b02c37daace98f33bb0567b59d1 Nov 03 22:29:15.835802 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:15.835750 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" event={"ID":"3611ea43-1944-4e72-aa7e-966a23812c7f","Type":"ContainerStarted","Data":"a26a9afd55c907b5dd0f3f943a66cae1b6a44b02c37daace98f33bb0567b59d1"} Nov 03 22:29:17.846408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:17.846374 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" event={"ID":"3611ea43-1944-4e72-aa7e-966a23812c7f","Type":"ContainerStarted","Data":"9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300"} Nov 03 22:29:17.846782 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:17.846567 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:17.970920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:17.970857 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" podStartSLOduration=1.977064982 podStartE2EDuration="3.970843089s" totalImagesPullingTime="1.993778107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:17.969123462 +0000 UTC m=+401.181892533" watchObservedRunningTime="2026-11-03 22:29:17.970843089 +0000 UTC m=+401.183612139" Nov 03 22:29:19.833472 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:29:19.833440 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:29:23.862803 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:23.862755 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:29.911643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:29.911610 4898 generic.go:356] "Generic (PLEG): container finished" podID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerID="4d7b86a45d0023aa6a293952300bac25ecdc62921c316170cb9f6057ff82b589" exitCode=0 Nov 03 22:29:29.912084 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:29.911683 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" event={"ID":"ff4a701c-59e5-49be-a92b-76cab43d4d90","Type":"ContainerDied","Data":"4d7b86a45d0023aa6a293952300bac25ecdc62921c316170cb9f6057ff82b589"} Nov 03 22:29:29.912084 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:29.912058 4898 scope.go:122] "RemoveContainer" containerID="4d7b86a45d0023aa6a293952300bac25ecdc62921c316170cb9f6057ff82b589" Nov 03 22:29:29.913166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:29.913132 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:29.913248 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:29.913205 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:29.913248 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:29.913237 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:30.919287 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:30.919247 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" event={"ID":"ff4a701c-59e5-49be-a92b-76cab43d4d90","Type":"ContainerStarted","Data":"a5e02f23aa879339b649c94d1c5c5710536d49a94379c1bf492284b36ce10530"} Nov 03 22:29:31.717960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:31.717922 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:29:31.719161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:31.719133 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-d5brv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:29:31.719263 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:31.719202 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 03 22:29:32.718302 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:32.718272 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-d5brv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:29:32.718672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:32.718336 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 03 22:29:33.717758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:33.717723 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-d5brv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:29:33.717938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:33.717812 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 03 22:29:34.717689 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:34.717650 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-d5brv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:29:34.718120 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:34.717716 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 03 22:29:34.913701 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:34.913671 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:29:35.718164 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:35.718123 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-d5brv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:29:35.718578 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:35.718190 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 03 22:29:35.993983 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:35.993892 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv"] Nov 03 22:29:35.994161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:35.994131 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" podUID="3611ea43-1944-4e72-aa7e-966a23812c7f" containerName="maas-api" containerID="cri-o://9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300" gracePeriod=30 Nov 03 22:29:36.237262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.237237 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:36.313511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.313446 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3611ea43-1944-4e72-aa7e-966a23812c7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611ea43-1944-4e72-aa7e-966a23812c7f-openshift-service-ca\") pod \"3611ea43-1944-4e72-aa7e-966a23812c7f\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " Nov 03 22:29:36.313511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.313486 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3611ea43-1944-4e72-aa7e-966a23812c7f-kube-api-access-pbjdd\" (UniqueName: \"kubernetes.io/projected/3611ea43-1944-4e72-aa7e-966a23812c7f-kube-api-access-pbjdd\") pod \"3611ea43-1944-4e72-aa7e-966a23812c7f\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " Nov 03 22:29:36.313695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.313573 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-maas-api-tls\") pod \"3611ea43-1944-4e72-aa7e-966a23812c7f\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " Nov 03 22:29:36.313695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.313596 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-metrics-certs\") pod \"3611ea43-1944-4e72-aa7e-966a23812c7f\" (UID: \"3611ea43-1944-4e72-aa7e-966a23812c7f\") " Nov 03 22:29:36.313899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.313874 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3611ea43-1944-4e72-aa7e-966a23812c7f-openshift-service-ca" pod "3611ea43-1944-4e72-aa7e-966a23812c7f" (UID: "3611ea43-1944-4e72-aa7e-966a23812c7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:36.315839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.315807 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-maas-api-tls" pod "3611ea43-1944-4e72-aa7e-966a23812c7f" (UID: "3611ea43-1944-4e72-aa7e-966a23812c7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:36.315967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.315874 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-metrics-certs" pod "3611ea43-1944-4e72-aa7e-966a23812c7f" (UID: "3611ea43-1944-4e72-aa7e-966a23812c7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:36.315967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.315886 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3611ea43-1944-4e72-aa7e-966a23812c7f-kube-api-access-pbjdd" pod "3611ea43-1944-4e72-aa7e-966a23812c7f" (UID: "3611ea43-1944-4e72-aa7e-966a23812c7f"). InnerVolumeSpecName "kube-api-access-pbjdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:36.415141 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.415109 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:36.415141 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.415135 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3611ea43-1944-4e72-aa7e-966a23812c7f-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:36.415141 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.415145 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611ea43-1944-4e72-aa7e-966a23812c7f-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:36.415354 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.415155 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjdd\" (UniqueName: \"kubernetes.io/projected/3611ea43-1944-4e72-aa7e-966a23812c7f-kube-api-access-pbjdd\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:36.721395 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.721364 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:29:36.722418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:36.722399 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:29:37.001106 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.001027 4898 generic.go:356] "Generic (PLEG): container finished" podID="3611ea43-1944-4e72-aa7e-966a23812c7f" containerID="9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300" exitCode=0 Nov 03 22:29:37.001278 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.001099 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" Nov 03 22:29:37.001278 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.001102 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" event={"ID":"3611ea43-1944-4e72-aa7e-966a23812c7f","Type":"ContainerDied","Data":"9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300"} Nov 03 22:29:37.001404 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.001297 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv" event={"ID":"3611ea43-1944-4e72-aa7e-966a23812c7f","Type":"ContainerDied","Data":"a26a9afd55c907b5dd0f3f943a66cae1b6a44b02c37daace98f33bb0567b59d1"} Nov 03 22:29:37.001404 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.001338 4898 scope.go:122] "RemoveContainer" containerID="9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300" Nov 03 22:29:37.012536 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.012507 4898 scope.go:122] "RemoveContainer" containerID="9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300" Nov 03 22:29:37.012869 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:29:37.012842 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300\": container with ID starting with 9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300 not found: ID does not exist" containerID="9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300" Nov 03 22:29:37.012965 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.012883 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300"} err="failed to get container status \"9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300\": rpc error: code = NotFound desc = could not find container \"9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300\": container with ID starting with 9ba7b32c66e7bba9b9f37e85612f21b620ee23009b18d1fbe87f71ae3e7b2300 not found: ID does not exist" Nov 03 22:29:37.146477 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.146439 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv"] Nov 03 22:29:37.150067 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:37.150041 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6nbfv"] Nov 03 22:29:38.904494 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:38.904459 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3611ea43-1944-4e72-aa7e-966a23812c7f" path="/var/lib/kubelet/pods/3611ea43-1944-4e72-aa7e-966a23812c7f/volumes" Nov 03 22:29:49.138098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.138060 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7vp25"] Nov 03 22:29:49.138504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.138465 4898 cpu_manager.go:414] "containerMap: removing container" podUID="3611ea43-1944-4e72-aa7e-966a23812c7f" containerName="maas-api" Nov 03 22:29:49.138504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.138478 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3611ea43-1944-4e72-aa7e-966a23812c7f" containerName="maas-api" Nov 03 22:29:49.138588 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.138574 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3611ea43-1944-4e72-aa7e-966a23812c7f" containerName="maas-api" Nov 03 22:29:49.140533 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.140517 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.198251 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.198220 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7vp25"] Nov 03 22:29:49.228534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.228498 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlnl\" (UniqueName: \"kubernetes.io/projected/665e97db-2cb5-44e4-950f-b8aae83c0df0-kube-api-access-cwlnl\") pod \"llmisvc-controller-manager-6466c554d8-7vp25\" (UID: \"665e97db-2cb5-44e4-950f-b8aae83c0df0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.228671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.228570 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/665e97db-2cb5-44e4-950f-b8aae83c0df0-cert\") pod \"llmisvc-controller-manager-6466c554d8-7vp25\" (UID: \"665e97db-2cb5-44e4-950f-b8aae83c0df0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.329884 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.329850 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/665e97db-2cb5-44e4-950f-b8aae83c0df0-cert\") pod \"llmisvc-controller-manager-6466c554d8-7vp25\" (UID: \"665e97db-2cb5-44e4-950f-b8aae83c0df0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.330040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.329923 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlnl\" (UniqueName: \"kubernetes.io/projected/665e97db-2cb5-44e4-950f-b8aae83c0df0-kube-api-access-cwlnl\") pod \"llmisvc-controller-manager-6466c554d8-7vp25\" (UID: \"665e97db-2cb5-44e4-950f-b8aae83c0df0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.332426 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.332397 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/665e97db-2cb5-44e4-950f-b8aae83c0df0-cert\") pod \"llmisvc-controller-manager-6466c554d8-7vp25\" (UID: \"665e97db-2cb5-44e4-950f-b8aae83c0df0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.391507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.391446 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlnl\" (UniqueName: \"kubernetes.io/projected/665e97db-2cb5-44e4-950f-b8aae83c0df0-kube-api-access-cwlnl\") pod \"llmisvc-controller-manager-6466c554d8-7vp25\" (UID: \"665e97db-2cb5-44e4-950f-b8aae83c0df0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.451901 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.451874 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:49.821541 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:49.821517 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7vp25"] Nov 03 22:29:49.823612 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:29:49.823582 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665e97db_2cb5_44e4_950f_b8aae83c0df0.slice/crio-a2644127e0be7b66ccbd487d1cae8d51593283a3e2b28e5074d74c748f45bb4d WatchSource:0}: Error finding container a2644127e0be7b66ccbd487d1cae8d51593283a3e2b28e5074d74c748f45bb4d: Status 404 returned error can't find the container with id a2644127e0be7b66ccbd487d1cae8d51593283a3e2b28e5074d74c748f45bb4d Nov 03 22:29:49.835286 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:29:49.835256 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:29:50.065867 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:50.065833 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" event={"ID":"665e97db-2cb5-44e4-950f-b8aae83c0df0","Type":"ContainerStarted","Data":"a2644127e0be7b66ccbd487d1cae8d51593283a3e2b28e5074d74c748f45bb4d"} Nov 03 22:29:51.072343 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:51.072307 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" event={"ID":"665e97db-2cb5-44e4-950f-b8aae83c0df0","Type":"ContainerStarted","Data":"02f05aafa3d3f2a9318102f16b469d66805aca7631030e0d9d1714535d73f5a1"} Nov 03 22:29:51.072734 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:51.072395 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:29:51.189235 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:29:51.189180 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" podStartSLOduration=1.419045908 podStartE2EDuration="2.189165765s" totalImagesPullingTime="770.119857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:51.188046944 +0000 UTC m=+434.400816015" watchObservedRunningTime="2026-11-03 22:29:51.189165765 +0000 UTC m=+434.401934862" Nov 03 22:30:19.834041 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:30:19.834001 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:30:22.079330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.079299 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7vp25" Nov 03 22:30:22.247899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.247854 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq"] Nov 03 22:30:22.248222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.248190 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" podUID="91f6791d-a54e-4662-91f7-c1edf645f7e4" containerName="manager" containerID="cri-o://1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2" gracePeriod=30 Nov 03 22:30:22.508971 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.508949 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:30:22.522289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.522262 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f6791d-a54e-4662-91f7-c1edf645f7e4-kube-api-access-zqqps\" (UniqueName: \"kubernetes.io/projected/91f6791d-a54e-4662-91f7-c1edf645f7e4-kube-api-access-zqqps\") pod \"91f6791d-a54e-4662-91f7-c1edf645f7e4\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " Nov 03 22:30:22.522408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.522313 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91f6791d-a54e-4662-91f7-c1edf645f7e4-cert\" (UniqueName: \"kubernetes.io/secret/91f6791d-a54e-4662-91f7-c1edf645f7e4-cert\") pod \"91f6791d-a54e-4662-91f7-c1edf645f7e4\" (UID: \"91f6791d-a54e-4662-91f7-c1edf645f7e4\") " Nov 03 22:30:22.524425 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.524395 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f6791d-a54e-4662-91f7-c1edf645f7e4-cert" pod "91f6791d-a54e-4662-91f7-c1edf645f7e4" (UID: "91f6791d-a54e-4662-91f7-c1edf645f7e4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:22.524543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.524521 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f6791d-a54e-4662-91f7-c1edf645f7e4-kube-api-access-zqqps" pod "91f6791d-a54e-4662-91f7-c1edf645f7e4" (UID: "91f6791d-a54e-4662-91f7-c1edf645f7e4"). InnerVolumeSpecName "kube-api-access-zqqps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:22.623510 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.623344 4898 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f6791d-a54e-4662-91f7-c1edf645f7e4-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:22.623510 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:22.623387 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqqps\" (UniqueName: \"kubernetes.io/projected/91f6791d-a54e-4662-91f7-c1edf645f7e4-kube-api-access-zqqps\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:23.221255 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.221169 4898 generic.go:356] "Generic (PLEG): container finished" podID="91f6791d-a54e-4662-91f7-c1edf645f7e4" containerID="1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2" exitCode=0 Nov 03 22:30:23.221255 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.221224 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" Nov 03 22:30:23.221255 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.221237 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" event={"ID":"91f6791d-a54e-4662-91f7-c1edf645f7e4","Type":"ContainerDied","Data":"1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2"} Nov 03 22:30:23.221735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.221277 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq" event={"ID":"91f6791d-a54e-4662-91f7-c1edf645f7e4","Type":"ContainerDied","Data":"73cb32c7720404fdd60991cc7fb508cf4a19bdd9da53849ddef127860029aeda"} Nov 03 22:30:23.221735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.221302 4898 scope.go:122] "RemoveContainer" containerID="1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2" Nov 03 22:30:23.229950 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.229931 4898 scope.go:122] "RemoveContainer" containerID="1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2" Nov 03 22:30:23.230224 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:30:23.230203 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2\": container with ID starting with 1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2 not found: ID does not exist" containerID="1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2" Nov 03 22:30:23.230269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.230234 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2"} err="failed to get container status \"1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2\": rpc error: code = NotFound desc = could not find container \"1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2\": container with ID starting with 1c24f2b9df743a0c3e6f1d12ae177b73030d8250354232d193500409928979b2 not found: ID does not exist" Nov 03 22:30:23.388134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.388098 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq"] Nov 03 22:30:23.438038 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:23.438005 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r6znq"] Nov 03 22:30:24.904461 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:30:24.904428 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f6791d-a54e-4662-91f7-c1edf645f7e4" path="/var/lib/kubelet/pods/91f6791d-a54e-4662-91f7-c1edf645f7e4/volumes" Nov 03 22:30:49.833772 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:30:49.833727 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:31:19.834710 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:31:19.834665 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:31:31.171611 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.171575 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct"] Nov 03 22:31:31.172053 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.172039 4898 cpu_manager.go:414] "containerMap: removing container" podUID="91f6791d-a54e-4662-91f7-c1edf645f7e4" containerName="manager" Nov 03 22:31:31.172097 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.172054 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f6791d-a54e-4662-91f7-c1edf645f7e4" containerName="manager" Nov 03 22:31:31.172139 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.172130 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f6791d-a54e-4662-91f7-c1edf645f7e4" containerName="manager" Nov 03 22:31:31.175557 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.175540 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.207900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.207863 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208078 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.207929 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208078 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.208047 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208203 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.208096 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208203 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.208131 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208304 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.208212 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5j8h\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-kube-api-access-v5j8h\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208304 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.208278 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208403 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.208306 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.208403 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.208337 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.282064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.282028 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct"] Nov 03 22:31:31.309632 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.309596 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.309856 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.309644 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.309856 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.309694 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310047 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.309859 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310047 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.309905 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5j8h\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-kube-api-access-v5j8h\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310093 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310122 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310191 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310381 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310357 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310391 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310430 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310537 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310461 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.310860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.310831 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.311254 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.311216 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.312563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.312536 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.312739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.312720 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.371541 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.371509 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.371664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.371634 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5j8h\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-kube-api-access-v5j8h\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j72ct\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.488926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.488823 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:31.922075 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.922049 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct"] Nov 03 22:31:31.923260 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:31:31.923232 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7d2e54_78db_45b1_8a8b_97fbd35f19a3.slice/crio-eff1ac122a0caeb5a524ffc0cffa03fe341c33e118775ee3ef24f73651060d62 WatchSource:0}: Error finding container eff1ac122a0caeb5a524ffc0cffa03fe341c33e118775ee3ef24f73651060d62: Status 404 returned error can't find the container with id eff1ac122a0caeb5a524ffc0cffa03fe341c33e118775ee3ef24f73651060d62 Nov 03 22:31:31.926667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.925701 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:31.926667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.925864 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:31.926667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:31.925962 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:32.568726 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:32.568684 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" event={"ID":"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3","Type":"ContainerStarted","Data":"313b5003e527a56e8d3d9a11a983d3ab8065351324f1e3bda474f803c32f12f6"} Nov 03 22:31:32.569154 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:32.568816 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" event={"ID":"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3","Type":"ContainerStarted","Data":"eff1ac122a0caeb5a524ffc0cffa03fe341c33e118775ee3ef24f73651060d62"} Nov 03 22:31:33.489957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:33.489923 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:33.491663 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:33.491632 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j72ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 22:31:33.491803 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:33.491698 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 22:31:34.489740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:34.489705 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j72ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 22:31:34.490142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:34.489773 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 22:31:35.489651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:35.489617 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j72ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 22:31:35.489832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:35.489671 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 22:31:36.490079 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:36.490045 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j72ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 22:31:36.490458 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:36.490107 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 22:31:37.489969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:37.489930 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j72ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 22:31:37.490163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:37.489986 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 22:31:38.489259 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:38.489221 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j72ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 22:31:38.489441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:38.489292 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 22:31:40.243870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:40.243836 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:40.244288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:40.244266 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:40.245187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:40.245166 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:31:40.450484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:31:40.450429 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podStartSLOduration=9.450416707 podStartE2EDuration="9.450416707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:32.748769954 +0000 UTC m=+535.961539021" watchObservedRunningTime="2026-11-03 22:31:40.450416707 +0000 UTC m=+543.663185757" Nov 03 22:31:49.833997 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:31:49.833955 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:32:19.834429 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:32:19.834351 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:32:36.963768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:36.963738 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:32:36.964271 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:36.964158 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:32:36.968721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:36.968698 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:32:36.968887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:36.968876 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:32:45.207268 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.207231 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq"] Nov 03 22:32:45.211029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.211008 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.267336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.267059 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ee74cb-openshift-default-dockercfg-zszk2\"" Nov 03 22:32:45.293986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.293951 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.293986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.293994 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.294234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.294118 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.294234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.294164 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.294234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.294211 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.294405 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.294240 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.294405 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.294306 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4eff3-033c-4bcf-9fcd-051294b437b3-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.294405 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.294339 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6sgq\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-kube-api-access-l6sgq\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.294405 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.294369 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.318703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.318672 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq"] Nov 03 22:32:45.395577 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395543 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.395758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395597 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.395758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395627 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.395758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395666 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.395758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395695 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.395758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395752 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4eff3-033c-4bcf-9fcd-051294b437b3-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395800 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6sgq\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-kube-api-access-l6sgq\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395848 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.395917 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.396026 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.396164 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.396242 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.396344 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.396583 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.396556 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4eff3-033c-4bcf-9fcd-051294b437b3-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.398195 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.398172 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.398424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.398405 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.457193 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.457155 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.457385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.457271 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6sgq\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-kube-api-access-l6sgq\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.527439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.527397 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:45.971656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.971631 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq"] Nov 03 22:32:45.973534 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:32:45.973503 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d4eff3_033c_4bcf_9fcd_051294b437b3.slice/crio-889e5b12df51d1d7a622d0e5b2757f17bf38d689a98125ab3bfbc77c31975012 WatchSource:0}: Error finding container 889e5b12df51d1d7a622d0e5b2757f17bf38d689a98125ab3bfbc77c31975012: Status 404 returned error can't find the container with id 889e5b12df51d1d7a622d0e5b2757f17bf38d689a98125ab3bfbc77c31975012 Nov 03 22:32:45.975711 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.975677 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:45.975804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.975741 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:45.975804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:45.975769 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:46.933659 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:46.933621 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" event={"ID":"65d4eff3-033c-4bcf-9fcd-051294b437b3","Type":"ContainerStarted","Data":"689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3"} Nov 03 22:32:46.934110 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:46.933676 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" event={"ID":"65d4eff3-033c-4bcf-9fcd-051294b437b3","Type":"ContainerStarted","Data":"889e5b12df51d1d7a622d0e5b2757f17bf38d689a98125ab3bfbc77c31975012"} Nov 03 22:32:47.110163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:47.110104 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" podStartSLOduration=2.110082924 podStartE2EDuration="2.110082924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:47.109550297 +0000 UTC m=+610.322319368" watchObservedRunningTime="2026-11-03 22:32:47.110082924 +0000 UTC m=+610.322851972" Nov 03 22:32:47.528032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:47.527997 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:47.532748 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:47.532717 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:47.938662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:47.938625 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:47.939646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:47.939622 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:32:49.837149 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:32:49.837111 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:32:57.793960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:57.793923 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd"] Nov 03 22:32:57.797544 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:57.797523 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:57.851433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:57.851386 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6f1190\"" Nov 03 22:32:57.901850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:57.901821 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6f1190-dockercfg-mlqw4\"" Nov 03 22:32:57.903610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:57.903585 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd"] Nov 03 22:32:57.915326 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:57.915297 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9687ae02-dd9b-4030-955a-1576db895e90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:57.915461 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:57.915340 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc84l\" (UniqueName: \"kubernetes.io/projected/9687ae02-dd9b-4030-955a-1576db895e90-kube-api-access-gc84l\") pod \"payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:58.016373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:58.016340 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc84l\" (UniqueName: \"kubernetes.io/projected/9687ae02-dd9b-4030-955a-1576db895e90-kube-api-access-gc84l\") pod \"payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:58.016531 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:58.016488 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9687ae02-dd9b-4030-955a-1576db895e90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:58.017204 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:58.017183 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9687ae02-dd9b-4030-955a-1576db895e90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:58.078202 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:58.078169 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc84l\" (UniqueName: \"kubernetes.io/projected/9687ae02-dd9b-4030-955a-1576db895e90-kube-api-access-gc84l\") pod \"payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:58.111320 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:58.111290 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:32:58.485190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:58.485158 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd"] Nov 03 22:32:58.488200 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:32:58.488154 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9687ae02_dd9b_4030_955a_1576db895e90.slice/crio-02220f0cef6b6080805c08baf380b80534cc8e20a8c093261ece5177b3373ad9 WatchSource:0}: Error finding container 02220f0cef6b6080805c08baf380b80534cc8e20a8c093261ece5177b3373ad9: Status 404 returned error can't find the container with id 02220f0cef6b6080805c08baf380b80534cc8e20a8c093261ece5177b3373ad9 Nov 03 22:32:58.490483 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:58.490462 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:32:59.002241 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:32:59.002201 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" event={"ID":"9687ae02-dd9b-4030-955a-1576db895e90","Type":"ContainerStarted","Data":"02220f0cef6b6080805c08baf380b80534cc8e20a8c093261ece5177b3373ad9"} Nov 03 22:33:00.089659 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.089617 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk"] Nov 03 22:33:00.093993 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.093964 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.147831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.147760 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9qhjs\"" Nov 03 22:33:00.198290 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.198254 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-ee74cb-serving-cert\"" Nov 03 22:33:00.198497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.198329 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:33:00.200700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.200672 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk"] Nov 03 22:33:00.239829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.237089 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.239829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.237135 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftk4\" (UniqueName: \"kubernetes.io/projected/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-kube-api-access-sftk4\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.239829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.237172 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-metrics-certs\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.239829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.237212 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.338463 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.338427 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-metrics-certs\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.338638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.338502 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.338638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.338600 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.338750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.338638 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftk4\" (UniqueName: \"kubernetes.io/projected/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-kube-api-access-sftk4\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.338850 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:00.338765 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-serving-cert: secret "maas-api-e2e-worker-w2-ee74cb-serving-cert" not found Nov 03 22:33:00.339036 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:00.338868 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls podName:670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa nodeName:}" failed. No retries permitted until 2026-11-03 22:33:00.838844829 +0000 UTC m=+624.051613875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls") pod "maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" (UID: "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa") : secret "maas-api-e2e-worker-w2-ee74cb-serving-cert" not found Nov 03 22:33:00.339331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.339312 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.341280 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.341219 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-metrics-certs\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.400848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.400806 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftk4\" (UniqueName: \"kubernetes.io/projected/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-kube-api-access-sftk4\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.843226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.843184 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:00.845994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:00.845965 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:01.010534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:01.010489 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:01.399825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:01.399776 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk"] Nov 03 22:33:01.404068 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:33:01.404031 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670a3d97_b6f5_4d5f_9d8c_5c1e678d0cfa.slice/crio-88746b4f636f267838a84998fc6ada0f530f206b92fa96ae1ee0408e4cbd810a WatchSource:0}: Error finding container 88746b4f636f267838a84998fc6ada0f530f206b92fa96ae1ee0408e4cbd810a: Status 404 returned error can't find the container with id 88746b4f636f267838a84998fc6ada0f530f206b92fa96ae1ee0408e4cbd810a Nov 03 22:33:02.021143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:02.021113 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" event={"ID":"9687ae02-dd9b-4030-955a-1576db895e90","Type":"ContainerStarted","Data":"41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea"} Nov 03 22:33:02.021335 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:02.021243 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:33:02.022410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:02.022385 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" event={"ID":"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa","Type":"ContainerStarted","Data":"88746b4f636f267838a84998fc6ada0f530f206b92fa96ae1ee0408e4cbd810a"} Nov 03 22:33:02.140542 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:02.140491 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" podStartSLOduration=2.275572814 podStartE2EDuration="5.14047723s" totalImagesPullingTime="2.864904416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:02.139848578 +0000 UTC m=+625.352617628" watchObservedRunningTime="2026-11-03 22:33:02.14047723 +0000 UTC m=+625.353246280" Nov 03 22:33:03.029077 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:03.029037 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" event={"ID":"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa","Type":"ContainerStarted","Data":"fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c"} Nov 03 22:33:03.029530 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:03.029294 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:03.148945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:03.148872 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" podStartSLOduration=2.251479071 podStartE2EDuration="3.148852917s" totalImagesPullingTime="897.373846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:03.148088723 +0000 UTC m=+626.360857800" watchObservedRunningTime="2026-11-03 22:33:03.148852917 +0000 UTC m=+626.361621968" Nov 03 22:33:04.997449 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:04.997415 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x"] Nov 03 22:33:05.001271 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.001247 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.057229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.057183 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-68fbd1-serving-cert\"" Nov 03 22:33:05.109577 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.109544 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x"] Nov 03 22:33:05.187486 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.187455 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgw2s\" (UniqueName: \"kubernetes.io/projected/1d915cf7-19ac-4e53-85d5-0b77359cdb93-kube-api-access-zgw2s\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.187671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.187521 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d915cf7-19ac-4e53-85d5-0b77359cdb93-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.187671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.187582 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-metrics-certs\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.187671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.187630 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-maas-api-tls\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.288907 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.288783 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d915cf7-19ac-4e53-85d5-0b77359cdb93-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.288907 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.288866 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-metrics-certs\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.288907 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.288903 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-maas-api-tls\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.289117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.288991 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgw2s\" (UniqueName: \"kubernetes.io/projected/1d915cf7-19ac-4e53-85d5-0b77359cdb93-kube-api-access-zgw2s\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.289452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.289429 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d915cf7-19ac-4e53-85d5-0b77359cdb93-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.291448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.291424 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-maas-api-tls\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.291552 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.291504 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-metrics-certs\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.350128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.350097 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgw2s\" (UniqueName: \"kubernetes.io/projected/1d915cf7-19ac-4e53-85d5-0b77359cdb93-kube-api-access-zgw2s\") pod \"maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.615613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.615581 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:05.992405 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:05.992362 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x"] Nov 03 22:33:05.994070 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:33:05.994037 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d915cf7_19ac_4e53_85d5_0b77359cdb93.slice/crio-47c7c3633e015be9c4073745e651c40db50b9760b15fd81b3d468bdeccc8c601 WatchSource:0}: Error finding container 47c7c3633e015be9c4073745e651c40db50b9760b15fd81b3d468bdeccc8c601: Status 404 returned error can't find the container with id 47c7c3633e015be9c4073745e651c40db50b9760b15fd81b3d468bdeccc8c601 Nov 03 22:33:06.045812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:06.045743 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" event={"ID":"1d915cf7-19ac-4e53-85d5-0b77359cdb93","Type":"ContainerStarted","Data":"47c7c3633e015be9c4073745e651c40db50b9760b15fd81b3d468bdeccc8c601"} Nov 03 22:33:07.053845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:07.053807 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" event={"ID":"1d915cf7-19ac-4e53-85d5-0b77359cdb93","Type":"ContainerStarted","Data":"6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d"} Nov 03 22:33:07.054231 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:07.053866 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:07.173178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:07.173068 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" podStartSLOduration=2.283037232 podStartE2EDuration="3.17305464s" totalImagesPullingTime="890.017408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:07.17187971 +0000 UTC m=+630.384648763" watchObservedRunningTime="2026-11-03 22:33:07.17305464 +0000 UTC m=+630.385823690" Nov 03 22:33:09.039524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:09.039483 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 22:33:13.032760 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:13.032724 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 22:33:13.066467 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:13.066434 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 22:33:15.213520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.213156 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495"] Nov 03 22:33:15.217501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.217476 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.270764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.270728 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-18c03a\"" Nov 03 22:33:15.270985 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.270880 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-18c03a-dockercfg-5pzzz\"" Nov 03 22:33:15.282300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.282270 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77czh\" (UniqueName: \"kubernetes.io/projected/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-kube-api-access-77czh\") pod \"payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.282438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.282376 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.323289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.323253 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495"] Nov 03 22:33:15.383114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.383076 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77czh\" (UniqueName: \"kubernetes.io/projected/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-kube-api-access-77czh\") pod \"payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.383276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.383222 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.384007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.383983 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.467809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.467671 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77czh\" (UniqueName: \"kubernetes.io/projected/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-kube-api-access-77czh\") pod \"payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.533698 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.533667 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:15.929912 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:15.929880 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495"] Nov 03 22:33:15.931570 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:33:15.931532 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5068bb_6e95_43f3_a4ef_e7f27fdca4da.slice/crio-a067f12abb269f5d1fc43b64159be6eb6e6f69b3b6b65e6c8620953f36e7d7cb WatchSource:0}: Error finding container a067f12abb269f5d1fc43b64159be6eb6e6f69b3b6b65e6c8620953f36e7d7cb: Status 404 returned error can't find the container with id a067f12abb269f5d1fc43b64159be6eb6e6f69b3b6b65e6c8620953f36e7d7cb Nov 03 22:33:16.107534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:16.107502 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" event={"ID":"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da","Type":"ContainerStarted","Data":"cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d"} Nov 03 22:33:16.107712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:16.107552 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" event={"ID":"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da","Type":"ContainerStarted","Data":"a067f12abb269f5d1fc43b64159be6eb6e6f69b3b6b65e6c8620953f36e7d7cb"} Nov 03 22:33:16.107712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:16.107628 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:19.863207 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:19.863117 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:33:24.151503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:24.151470 4898 generic.go:356] "Generic (PLEG): container finished" podID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerID="689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3" exitCode=0 Nov 03 22:33:24.152027 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:24.151543 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" event={"ID":"65d4eff3-033c-4bcf-9fcd-051294b437b3","Type":"ContainerDied","Data":"689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3"} Nov 03 22:33:24.152027 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:24.151937 4898 scope.go:122] "RemoveContainer" containerID="689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3" Nov 03 22:33:24.153284 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:24.153257 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:24.153362 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:24.153320 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:24.153362 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:24.153352 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:24.308259 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:24.306309 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" podStartSLOduration=9.306292991 podStartE2EDuration="9.306292991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:16.241302705 +0000 UTC m=+639.454071773" watchObservedRunningTime="2026-11-03 22:33:24.306292991 +0000 UTC m=+647.519062048" Nov 03 22:33:25.168498 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:25.168459 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" event={"ID":"65d4eff3-033c-4bcf-9fcd-051294b437b3","Type":"ContainerStarted","Data":"e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242"} Nov 03 22:33:25.528469 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:25.528386 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:33:25.528623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:25.528604 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:33:25.528672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:25.528652 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:33:26.528123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:26.528084 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:33:26.528507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:26.528139 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:33:27.117365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:27.117329 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 22:33:27.528623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:27.528541 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:33:27.528623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:27.528594 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:33:28.528230 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.528197 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:33:28.528383 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.528248 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:33:28.647812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.647295 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz"] Nov 03 22:33:28.663921 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.661363 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.703464 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.703420 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz"] Nov 03 22:33:28.708878 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.708831 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.709034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.708897 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-metrics-certs\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.709109 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.709034 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8l7q\" (UniqueName: \"kubernetes.io/projected/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-kube-api-access-f8l7q\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.709166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.709102 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.714349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.714325 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-18c03a-serving-cert\"" Nov 03 22:33:28.810008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.809931 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.810008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.809972 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-metrics-certs\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.810008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.810005 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8l7q\" (UniqueName: \"kubernetes.io/projected/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-kube-api-access-f8l7q\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.810287 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.810042 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.810287 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:28.810108 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-serving-cert: secret "maas-api-e2e-worker-w4-18c03a-serving-cert" not found Nov 03 22:33:28.810287 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:28.810229 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls podName:cc8494eb-5c7c-4df2-886a-eb4f8e4385b9 nodeName:}" failed. No retries permitted until 2026-11-03 22:33:29.310207048 +0000 UTC m=+652.522976081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls") pod "maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" (UID: "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9") : secret "maas-api-e2e-worker-w4-18c03a-serving-cert" not found Nov 03 22:33:28.810798 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.810767 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.812714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.812690 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-metrics-certs\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:28.871920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:28.871890 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8l7q\" (UniqueName: \"kubernetes.io/projected/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-kube-api-access-f8l7q\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:29.316232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:29.316195 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:29.318686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:29.318657 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:29.527740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:29.527708 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:33:29.527917 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:29.527802 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:33:29.576237 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:29.576158 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:29.956862 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:29.953397 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz"] Nov 03 22:33:29.963329 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:33:29.963297 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8494eb_5c7c_4df2_886a_eb4f8e4385b9.slice/crio-2030c88af75dcbd1a0a7416371e498b64feca74132c2de1de930aa247c6c43f2 WatchSource:0}: Error finding container 2030c88af75dcbd1a0a7416371e498b64feca74132c2de1de930aa247c6c43f2: Status 404 returned error can't find the container with id 2030c88af75dcbd1a0a7416371e498b64feca74132c2de1de930aa247c6c43f2 Nov 03 22:33:30.195220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:30.195180 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" event={"ID":"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9","Type":"ContainerStarted","Data":"2030c88af75dcbd1a0a7416371e498b64feca74132c2de1de930aa247c6c43f2"} Nov 03 22:33:30.532846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:30.532815 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:33:30.533130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:30.533110 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:33:30.533996 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:30.533971 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 22:33:31.202343 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:31.202303 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" event={"ID":"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9","Type":"ContainerStarted","Data":"b3b515e8556e1e60fece1606aeb21272c87c1a071c69d7dd77a40e9042fbb80a"} Nov 03 22:33:31.202754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:31.202578 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:31.320226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:31.320171 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" podStartSLOduration=2.501092012 podStartE2EDuration="3.320157637s" totalImagesPullingTime="819.065625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:31.319655149 +0000 UTC m=+654.532424219" watchObservedRunningTime="2026-11-03 22:33:31.320157637 +0000 UTC m=+654.532926706" Nov 03 22:33:33.491279 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.491230 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb"] Nov 03 22:33:33.497392 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.497362 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.551436 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.551402 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb"] Nov 03 22:33:33.559432 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.559404 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.559554 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.559445 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-metrics-certs\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.559595 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.559548 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjm5\" (UniqueName: \"kubernetes.io/projected/82a88795-e98d-4780-b672-3a6dee25b4d3-kube-api-access-tpjm5\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.559595 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.559579 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a88795-e98d-4780-b672-3a6dee25b4d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.599305 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.599268 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-545b5d-serving-cert\"" Nov 03 22:33:33.660621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.660589 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.660779 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.660628 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-metrics-certs\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.660779 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.660667 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjm5\" (UniqueName: \"kubernetes.io/projected/82a88795-e98d-4780-b672-3a6dee25b4d3-kube-api-access-tpjm5\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.660779 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.660686 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a88795-e98d-4780-b672-3a6dee25b4d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.660779 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:33.660744 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-serving-cert: secret "maas-api-e2e-worker-w1-545b5d-serving-cert" not found Nov 03 22:33:33.660963 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:33.660870 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls podName:82a88795-e98d-4780-b672-3a6dee25b4d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:33:34.160849228 +0000 UTC m=+657.373618272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls") pod "maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" (UID: "82a88795-e98d-4780-b672-3a6dee25b4d3") : secret "maas-api-e2e-worker-w1-545b5d-serving-cert" not found Nov 03 22:33:33.661517 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.661467 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a88795-e98d-4780-b672-3a6dee25b4d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.663235 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.663213 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-metrics-certs\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:33.722314 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:33.722288 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjm5\" (UniqueName: \"kubernetes.io/projected/82a88795-e98d-4780-b672-3a6dee25b4d3-kube-api-access-tpjm5\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:34.166993 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:34.166951 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:34.169349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:34.169322 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:34.410973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:34.410942 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:34.790571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:34.790540 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb"] Nov 03 22:33:34.792740 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:33:34.792713 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a88795_e98d_4780_b672_3a6dee25b4d3.slice/crio-efb242275650fdfdfc479330cdcd667a0e5ee8d2680bd881ba10b85dd04b8c1f WatchSource:0}: Error finding container efb242275650fdfdfc479330cdcd667a0e5ee8d2680bd881ba10b85dd04b8c1f: Status 404 returned error can't find the container with id efb242275650fdfdfc479330cdcd667a0e5ee8d2680bd881ba10b85dd04b8c1f Nov 03 22:33:35.223390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:35.223348 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" event={"ID":"82a88795-e98d-4780-b672-3a6dee25b4d3","Type":"ContainerStarted","Data":"efb242275650fdfdfc479330cdcd667a0e5ee8d2680bd881ba10b85dd04b8c1f"} Nov 03 22:33:36.230753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:36.230716 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" event={"ID":"82a88795-e98d-4780-b672-3a6dee25b4d3","Type":"ContainerStarted","Data":"4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893"} Nov 03 22:33:36.231151 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:36.230850 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:36.352615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:36.352547 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" podStartSLOduration=2.450799164 podStartE2EDuration="3.352528781s" totalImagesPullingTime="901.729617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:36.351593843 +0000 UTC m=+659.564362893" watchObservedRunningTime="2026-11-03 22:33:36.352528781 +0000 UTC m=+659.565297834" Nov 03 22:33:37.213077 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:37.213052 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 22:33:42.241385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:33:42.241351 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 22:33:49.834462 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:33:49.834423 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:34:19.834069 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:34:19.834023 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:34:49.833696 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:34:49.833659 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:35:19.834710 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:35:19.834627 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:35:46.154709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.154677 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2"] Nov 03 22:35:46.158567 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.158546 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.228848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.228813 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2"] Nov 03 22:35:46.262754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.262723 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"kube-root-ca.crt\"" Nov 03 22:35:46.262918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.262844 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"openshift-service-ca.crt\"" Nov 03 22:35:46.262918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.262858 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"premium-w6-5e72ad-kserve-self-signed-certs\"" Nov 03 22:35:46.262918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.262881 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"default-dockercfg-pkpfl\"" Nov 03 22:35:46.264416 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.264395 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-dshm\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.264554 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.264428 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d8093ff-1802-42b4-bf19-449e104baf04-tls-certs\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.264554 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.264454 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89prd\" (UniqueName: \"kubernetes.io/projected/3d8093ff-1802-42b4-bf19-449e104baf04-kube-api-access-89prd\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.264554 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.264529 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-home\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.264692 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.264569 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-tmp-dir\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.264728 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.264697 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-model-cache\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.365547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.365513 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-model-cache\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.365731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.365591 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-dshm\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.365731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.365614 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d8093ff-1802-42b4-bf19-449e104baf04-tls-certs\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.365731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.365639 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89prd\" (UniqueName: \"kubernetes.io/projected/3d8093ff-1802-42b4-bf19-449e104baf04-kube-api-access-89prd\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.365731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.365659 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-home\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.365731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.365700 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-tmp-dir\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.366113 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.366015 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-model-cache\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.366113 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.366089 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-home\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.366205 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.366143 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-tmp-dir\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.368015 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.367994 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-dshm\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.368138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.368104 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d8093ff-1802-42b4-bf19-449e104baf04-tls-certs\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.472887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.472807 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89prd\" (UniqueName: \"kubernetes.io/projected/3d8093ff-1802-42b4-bf19-449e104baf04-kube-api-access-89prd\") pod \"premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.478436 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.478407 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:46.877369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.877338 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2"] Nov 03 22:35:46.878152 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:35:46.878118 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d8093ff_1802_42b4_bf19_449e104baf04.slice/crio-39757cf6de4d229650911a030bcdf8daeef79166ad2fae20814bec2a3715d97c WatchSource:0}: Error finding container 39757cf6de4d229650911a030bcdf8daeef79166ad2fae20814bec2a3715d97c: Status 404 returned error can't find the container with id 39757cf6de4d229650911a030bcdf8daeef79166ad2fae20814bec2a3715d97c Nov 03 22:35:46.939776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:46.939741 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" event={"ID":"3d8093ff-1802-42b4-bf19-449e104baf04","Type":"ContainerStarted","Data":"39757cf6de4d229650911a030bcdf8daeef79166ad2fae20814bec2a3715d97c"} Nov 03 22:35:48.953420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:48.953384 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" event={"ID":"3d8093ff-1802-42b4-bf19-449e104baf04","Type":"ContainerStarted","Data":"4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e"} Nov 03 22:35:49.071371 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:49.071323 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podStartSLOduration=1.508177344 podStartE2EDuration="3.071309319s" totalImagesPullingTime="1.563131975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:49.070360938 +0000 UTC m=+792.283129990" watchObservedRunningTime="2026-11-03 22:35:49.071309319 +0000 UTC m=+792.284078402" Nov 03 22:35:49.478885 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:49.478853 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:49.834807 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:35:49.834746 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:35:56.479299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:56.479260 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:56.492392 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:56.492367 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:35:57.007007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:35:57.006979 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:36:14.980007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:14.979963 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b"] Nov 03 22:36:14.983835 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:14.983813 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.051781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.051746 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"openshift-service-ca.crt\"" Nov 03 22:36:15.055237 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.055204 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b"] Nov 03 22:36:15.087733 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.087701 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"kube-root-ca.crt\"" Nov 03 22:36:15.087890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.087705 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"premium-w4-18c03a-kserve-self-signed-certs\"" Nov 03 22:36:15.087890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.087762 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"default-dockercfg-6xfnv\"" Nov 03 22:36:15.151379 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.151337 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f23050-48f9-4b70-b3e5-0794b749e533-tls-certs\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.151563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.151390 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-home\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.151563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.151517 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbh2f\" (UniqueName: \"kubernetes.io/projected/d6f23050-48f9-4b70-b3e5-0794b749e533-kube-api-access-kbh2f\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.151683 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.151665 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-dshm\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.151737 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.151692 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-tmp-dir\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.151780 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.151754 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-model-cache\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.252654 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.252560 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f23050-48f9-4b70-b3e5-0794b749e533-tls-certs\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.252654 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.252614 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-home\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.252654 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.252645 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbh2f\" (UniqueName: \"kubernetes.io/projected/d6f23050-48f9-4b70-b3e5-0794b749e533-kube-api-access-kbh2f\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.252974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.252756 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-dshm\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.252974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.252779 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-tmp-dir\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.252974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.252833 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-model-cache\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.253292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.253267 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-model-cache\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.253551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.253515 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-tmp-dir\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.253636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.253571 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-home\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.260118 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.260093 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-dshm\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.260369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.260349 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f23050-48f9-4b70-b3e5-0794b749e533-tls-certs\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.314488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.314459 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbh2f\" (UniqueName: \"kubernetes.io/projected/d6f23050-48f9-4b70-b3e5-0794b749e533-kube-api-access-kbh2f\") pod \"premium-w4-18c03a-kserve-7787df7cf6-wkr4b\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.596420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.596382 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:15.970643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:15.970621 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b"] Nov 03 22:36:15.973411 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:36:15.973377 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6f23050_48f9_4b70_b3e5_0794b749e533.slice/crio-6a370042e34da8698fc7271342bc6991ed8f14e6a2533c413b485e776040a7e6 WatchSource:0}: Error finding container 6a370042e34da8698fc7271342bc6991ed8f14e6a2533c413b485e776040a7e6: Status 404 returned error can't find the container with id 6a370042e34da8698fc7271342bc6991ed8f14e6a2533c413b485e776040a7e6 Nov 03 22:36:16.097931 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:16.097901 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" event={"ID":"d6f23050-48f9-4b70-b3e5-0794b749e533","Type":"ContainerStarted","Data":"c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa"} Nov 03 22:36:16.098304 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:16.097939 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" event={"ID":"d6f23050-48f9-4b70-b3e5-0794b749e533","Type":"ContainerStarted","Data":"6a370042e34da8698fc7271342bc6991ed8f14e6a2533c413b485e776040a7e6"} Nov 03 22:36:16.215286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:16.215229 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podStartSLOduration=2.215214843 podStartE2EDuration="2.215214843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:16.214677407 +0000 UTC m=+819.427446458" watchObservedRunningTime="2026-11-03 22:36:16.215214843 +0000 UTC m=+819.427983885" Nov 03 22:36:16.597169 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:16.597129 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:19.835229 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:36:19.835190 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:36:25.597275 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:25.597229 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:25.610315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:25.610286 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:26.175483 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:26.175454 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:36:30.707712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.707672 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt"] Nov 03 22:36:30.713610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.713582 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.768365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.768328 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6f1190-openshift-default-dockercfg-fjmzm\"" Nov 03 22:36:30.809309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809271 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809506 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809319 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809506 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809409 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809506 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809454 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809517 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809546 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809582 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809633 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwvp\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-kube-api-access-xzwvp\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.809809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.809690 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df6004b-a782-479f-9966-9c38ad4ed0a7-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.820724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.820692 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt"] Nov 03 22:36:30.910118 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910089 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910132 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910242 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910279 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910494 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910312 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910494 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910347 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910494 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910470 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910506 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwvp\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-kube-api-access-xzwvp\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910576 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910766 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910670 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910865 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910772 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910898 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.910927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.910919 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df6004b-a782-479f-9966-9c38ad4ed0a7-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.911432 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.911412 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df6004b-a782-479f-9966-9c38ad4ed0a7-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.912992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.912964 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.913103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.912989 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.971724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.971654 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:30.971724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:30.971694 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwvp\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-kube-api-access-xzwvp\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:31.048495 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:31.048466 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:31.480764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:31.480737 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt"] Nov 03 22:36:31.480901 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:36:31.480877 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df6004b_a782_479f_9966_9c38ad4ed0a7.slice/crio-9e780cb841661f946e45f43f24a5087ca18f8543c980be0d420c3fcbf1568c9e WatchSource:0}: Error finding container 9e780cb841661f946e45f43f24a5087ca18f8543c980be0d420c3fcbf1568c9e: Status 404 returned error can't find the container with id 9e780cb841661f946e45f43f24a5087ca18f8543c980be0d420c3fcbf1568c9e Nov 03 22:36:31.483441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:31.483402 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:36:31.483531 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:31.483482 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:36:31.483531 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:31.483525 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:36:32.198689 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:32.198652 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" event={"ID":"7df6004b-a782-479f-9966-9c38ad4ed0a7","Type":"ContainerStarted","Data":"bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45"} Nov 03 22:36:32.199129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:32.198702 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" event={"ID":"7df6004b-a782-479f-9966-9c38ad4ed0a7","Type":"ContainerStarted","Data":"9e780cb841661f946e45f43f24a5087ca18f8543c980be0d420c3fcbf1568c9e"} Nov 03 22:36:32.327013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:32.326963 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" podStartSLOduration=2.326949647 podStartE2EDuration="2.326949647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:32.324687387 +0000 UTC m=+835.537456438" watchObservedRunningTime="2026-11-03 22:36:32.326949647 +0000 UTC m=+835.539718698" Nov 03 22:36:33.053010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:33.052545 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:33.053010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:33.052673 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 22:36:33.053010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:33.052715 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 22:36:34.050813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:34.050135 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 22:36:34.050813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:34.050196 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 22:36:35.055813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:35.055203 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 22:36:35.055813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:35.055264 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 22:36:36.049813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:36.049223 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 22:36:36.049813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:36.049290 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 22:36:37.050223 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:37.050179 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 22:36:37.050868 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:37.050833 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 22:36:38.907622 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:38.907589 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:38.908155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:38.908030 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:38.909030 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:36:38.909008 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:36:49.834728 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:36:49.834681 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:37:19.834448 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:37:19.834410 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:37:31.759274 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.759242 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk"] Nov 03 22:37:31.763117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.763099 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.816062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.816028 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-545b5d-openshift-default-dockercfg-t5wb5\"" Nov 03 22:37:31.855040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855002 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bba430-98f2-4ab6-a07f-a942ba4ad331-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855045 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5g2\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-kube-api-access-sq5g2\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855065 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855173 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855202 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855219 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855301 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855432 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.855543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.855460 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.868576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.868547 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk"] Nov 03 22:37:31.956032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.955994 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956027 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956090 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bba430-98f2-4ab6-a07f-a942ba4ad331-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956116 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5g2\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-kube-api-access-sq5g2\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956136 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956157 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956191 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956220 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956253 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956525 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956649 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956683 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956838 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956729 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.956974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.956955 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bba430-98f2-4ab6-a07f-a942ba4ad331-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.958594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.958574 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:31.958731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:31.958713 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:32.019029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.018948 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:32.019239 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.019219 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5g2\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-kube-api-access-sq5g2\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:32.076635 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.076603 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:32.505074 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.505042 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk"] Nov 03 22:37:32.506279 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:37:32.506253 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bba430_98f2_4ab6_a07f_a942ba4ad331.slice/crio-15b46650589769f8acac91b61da5f400e918605a8d224c83667b0afb371890d3 WatchSource:0}: Error finding container 15b46650589769f8acac91b61da5f400e918605a8d224c83667b0afb371890d3: Status 404 returned error can't find the container with id 15b46650589769f8acac91b61da5f400e918605a8d224c83667b0afb371890d3 Nov 03 22:37:32.508553 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.508519 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:32.508642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.508580 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:32.508642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.508607 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:32.538671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:32.538645 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" event={"ID":"30bba430-98f2-4ab6-a07f-a942ba4ad331","Type":"ContainerStarted","Data":"15b46650589769f8acac91b61da5f400e918605a8d224c83667b0afb371890d3"} Nov 03 22:37:33.566273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:33.566235 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" event={"ID":"30bba430-98f2-4ab6-a07f-a942ba4ad331","Type":"ContainerStarted","Data":"ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d"} Nov 03 22:37:33.688648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:33.688595 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podStartSLOduration=2.6885800079999997 podStartE2EDuration="2.688580008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:33.686640178 +0000 UTC m=+896.899409250" watchObservedRunningTime="2026-11-03 22:37:33.688580008 +0000 UTC m=+896.901349056" Nov 03 22:37:34.077442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:34.077410 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:34.078664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:34.078640 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:37:34.078828 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:34.078683 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 22:37:35.077096 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:35.077060 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:37:35.077579 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:35.077114 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 22:37:36.077410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:36.077372 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:37:36.077912 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:36.077440 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 22:37:37.020175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:37.020136 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:37:37.020473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:37.020451 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:37:37.025373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:37.025351 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:37:37.026070 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:37.026042 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:37:37.077346 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:37.077307 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:37:37.077516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:37.077364 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 22:37:38.077527 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:38.077485 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:37:38.078037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:38.077545 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 22:37:39.162202 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:39.162140 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:39.162671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:39.162611 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:39.163529 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:39.163510 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:37:49.014340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.014305 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x"] Nov 03 22:37:49.019406 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.019381 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.065577 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.065549 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj"] Nov 03 22:37:49.069386 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.069368 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.118466 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.118431 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x"] Nov 03 22:37:49.119947 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.119915 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj"] Nov 03 22:37:49.135465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135432 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c9b845-2337-4d01-8b70-41443825c175-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.135641 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135487 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.135641 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135522 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.135719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135657 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c9b845-2337-4d01-8b70-41443825c175-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.135761 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135715 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfp6\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-kube-api-access-dtfp6\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.135861 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135837 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbtw\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-kube-api-access-bpbtw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.135922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135902 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.135966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135945 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0add3bd-313c-4c30-a50c-5639957343e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.136012 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.135977 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0add3bd-313c-4c30-a50c-5639957343e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.136053 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136041 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.136121 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136105 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.136223 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136203 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.136276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136260 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.136366 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136319 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.136411 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136373 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.136451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136439 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.136485 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136472 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.136525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.136498 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.237535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.237502 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.237694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.237560 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.237694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.237601 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.237694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.237647 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.237694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.237672 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.237913 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.237695 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.237913 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.237733 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238002 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238115 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238084 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238112 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238120 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238169 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238186 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238205 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238222 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c9b845-2337-4d01-8b70-41443825c175-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238262 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238297 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238344 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c9b845-2337-4d01-8b70-41443825c175-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238382 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfp6\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-kube-api-access-dtfp6\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238417 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbtw\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-kube-api-access-bpbtw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.238502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238468 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238494 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0add3bd-313c-4c30-a50c-5639957343e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238526 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0add3bd-313c-4c30-a50c-5639957343e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238583 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238637 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.238944 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.238912 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.239350 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.239326 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0add3bd-313c-4c30-a50c-5639957343e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.239460 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.239355 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c9b845-2337-4d01-8b70-41443825c175-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.240170 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.240145 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.240584 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.240565 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.240814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.240761 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c9b845-2337-4d01-8b70-41443825c175-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.241494 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.241475 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0add3bd-313c-4c30-a50c-5639957343e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.300024 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.299931 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.300024 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.299954 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbtw\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-kube-api-access-bpbtw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.300196 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.300023 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfp6\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-kube-api-access-dtfp6\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4vg6x\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.300196 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.300034 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ftxcj\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.332514 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.332488 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:49.385029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.384991 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:49.784706 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.783571 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x"] Nov 03 22:37:49.785875 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.785800 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.785988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.785926 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.786050 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.786007 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.829826 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.829414 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj"] Nov 03 22:37:49.834105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.833672 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.834105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.833764 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.834105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:49.833873 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.840109 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:37:49.840080 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:37:50.674504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:50.674464 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" event={"ID":"a0add3bd-313c-4c30-a50c-5639957343e1","Type":"ContainerStarted","Data":"24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690"} Nov 03 22:37:50.674986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:50.674531 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" event={"ID":"a0add3bd-313c-4c30-a50c-5639957343e1","Type":"ContainerStarted","Data":"aac8975d9d6d229f43c758c95a444357f49cda044fd89d79a1255c8b0181d0c7"} Nov 03 22:37:50.676500 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:50.676467 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" event={"ID":"11c9b845-2337-4d01-8b70-41443825c175","Type":"ContainerStarted","Data":"da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c"} Nov 03 22:37:50.676639 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:50.676525 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" event={"ID":"11c9b845-2337-4d01-8b70-41443825c175","Type":"ContainerStarted","Data":"14a44a7af6ddc0c1284e6457620afd400ba61c3d7831a63a0ce6249f4911407d"} Nov 03 22:37:50.853608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:50.853535 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podStartSLOduration=2.853515153 podStartE2EDuration="2.853515153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:50.851689148 +0000 UTC m=+914.064458221" watchObservedRunningTime="2026-11-03 22:37:50.853515153 +0000 UTC m=+914.066284205" Nov 03 22:37:50.979869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:50.978963 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podStartSLOduration=2.9789411599999998 podStartE2EDuration="2.97894116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:50.976357708 +0000 UTC m=+914.189126759" watchObservedRunningTime="2026-11-03 22:37:50.97894116 +0000 UTC m=+914.191710214" Nov 03 22:37:51.333616 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:51.333586 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:51.341814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:51.337913 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4vg6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:37:51.341814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:51.337971 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 22:37:51.392579 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:51.392543 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:51.392985 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:51.392960 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ftxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:37:51.393189 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:51.393158 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:37:52.333310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:52.333272 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4vg6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:37:52.333841 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:52.333333 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 22:37:52.424812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:52.424637 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ftxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:37:52.424812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:52.424705 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:37:53.333680 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:53.333635 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4vg6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:37:53.334194 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:53.333700 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 22:37:53.386230 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:53.386189 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ftxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:37:53.386515 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:53.386480 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:37:54.333234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:54.333199 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4vg6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:37:54.333412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:54.333253 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 22:37:54.387940 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:54.387898 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ftxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:37:54.388560 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:54.388523 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:37:55.333219 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:55.333184 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4vg6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:37:55.333395 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:55.333235 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 22:37:55.385971 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:55.385933 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ftxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:37:55.386225 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:55.386198 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:37:56.333142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:56.333103 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4vg6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:37:56.333525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:56.333170 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 22:37:56.386615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:56.386578 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ftxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:37:56.386774 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:56.386642 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:37:57.333932 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:57.333893 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4vg6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:37:57.334327 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:57.333961 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 22:37:57.386470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:57.386431 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ftxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:37:57.386626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:57.386499 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:37:58.871292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:58.871259 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:58.873264 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:58.873230 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:58.874074 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:58.874052 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:37:58.908757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:58.908722 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:58.909362 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:58.909332 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:37:58.910203 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:37:58.910184 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:38:19.835471 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:38:19.835430 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:38:49.834523 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:38:49.834488 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:38:54.479585 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.479539 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9"] Nov 03 22:38:54.483778 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.483752 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.539333 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.539296 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9"] Nov 03 22:38:54.587068 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.587034 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6f1190\"/\"openshift-service-ca.crt\"" Nov 03 22:38:54.587313 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.587133 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6f1190\"/\"kube-root-ca.crt\"" Nov 03 22:38:54.587313 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.587190 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6f1190\"/\"distinct-w5-6f1190-kserve-self-signed-certs\"" Nov 03 22:38:54.587313 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.587253 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6f1190\"/\"default-dockercfg-k9b9h\"" Nov 03 22:38:54.643938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.643739 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-home\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.643938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.643856 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-tmp-dir\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.643938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.643890 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-dshm\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.644249 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.644006 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9gb\" (UniqueName: \"kubernetes.io/projected/34711dce-3ebd-4870-9431-f06db08ee855-kube-api-access-6s9gb\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.644249 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.644109 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-model-cache\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.644249 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.644193 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34711dce-3ebd-4870-9431-f06db08ee855-tls-certs\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.745735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.745627 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34711dce-3ebd-4870-9431-f06db08ee855-tls-certs\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.745937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.745808 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-home\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.745937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.745848 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-tmp-dir\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.745937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.745883 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-dshm\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.746103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.745933 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9gb\" (UniqueName: \"kubernetes.io/projected/34711dce-3ebd-4870-9431-f06db08ee855-kube-api-access-6s9gb\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.746103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.745976 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-model-cache\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.746242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.746214 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-home\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.746343 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.746321 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-model-cache\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.746917 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.746888 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-tmp-dir\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.748930 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.748903 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34711dce-3ebd-4870-9431-f06db08ee855-tls-certs\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.750021 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.749998 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-dshm\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:54.807944 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:54.807905 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9gb\" (UniqueName: \"kubernetes.io/projected/34711dce-3ebd-4870-9431-f06db08ee855-kube-api-access-6s9gb\") pod \"distinct-w5-6f1190-kserve-df5d8fc95-r7tj9\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:55.096571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:55.096537 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:55.482777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:55.482751 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9"] Nov 03 22:38:55.497886 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:38:55.497857 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34711dce_3ebd_4870_9431_f06db08ee855.slice/crio-f1808cb6930bf4bd258cdc03772263f17d067b0d972df11b2c1334508328b265 WatchSource:0}: Error finding container f1808cb6930bf4bd258cdc03772263f17d067b0d972df11b2c1334508328b265: Status 404 returned error can't find the container with id f1808cb6930bf4bd258cdc03772263f17d067b0d972df11b2c1334508328b265 Nov 03 22:38:56.124323 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:56.124283 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" event={"ID":"34711dce-3ebd-4870-9431-f06db08ee855","Type":"ContainerStarted","Data":"66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884"} Nov 03 22:38:56.124323 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:56.124333 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" event={"ID":"34711dce-3ebd-4870-9431-f06db08ee855","Type":"ContainerStarted","Data":"f1808cb6930bf4bd258cdc03772263f17d067b0d972df11b2c1334508328b265"} Nov 03 22:38:56.125950 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:56.125922 4898 generic.go:356] "Generic (PLEG): container finished" podID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerID="0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6" exitCode=1 Nov 03 22:38:56.126074 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:56.125999 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-p66vm" event={"ID":"f7ff4bb6-8c87-4027-baf5-0a80a0b44261","Type":"ContainerDied","Data":"0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6"} Nov 03 22:38:56.126441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:56.126424 4898 scope.go:122] "RemoveContainer" containerID="0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6" Nov 03 22:38:56.126844 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:56.126828 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:38:56.241645 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:56.241597 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podStartSLOduration=2.241583073 podStartE2EDuration="2.241583073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:56.241164238 +0000 UTC m=+979.453933307" watchObservedRunningTime="2026-11-03 22:38:56.241583073 +0000 UTC m=+979.454352123" Nov 03 22:38:57.097072 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:57.097032 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:38:57.119834 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:57.119782 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:38:57.134420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:57.134393 4898 generic.go:356] "Generic (PLEG): container finished" podID="3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e" containerID="0ad5c8080ffddc6abdcaa993b066dc32507595c0949e8b06447b6bc24c7df572" exitCode=1 Nov 03 22:38:57.134501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:57.134443 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" event={"ID":"3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e","Type":"ContainerDied","Data":"0ad5c8080ffddc6abdcaa993b066dc32507595c0949e8b06447b6bc24c7df572"} Nov 03 22:38:57.135076 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:57.135054 4898 scope.go:122] "RemoveContainer" containerID="0ad5c8080ffddc6abdcaa993b066dc32507595c0949e8b06447b6bc24c7df572" Nov 03 22:38:58.142683 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:58.142642 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" event={"ID":"3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e","Type":"ContainerStarted","Data":"eac1ee4914f4d3fccdba593869d17c3bb02ee76e16464e3dcad435b60c8d25d0"} Nov 03 22:38:58.143182 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:58.142874 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:38:58.144832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:58.144771 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-p66vm" event={"ID":"f7ff4bb6-8c87-4027-baf5-0a80a0b44261","Type":"ContainerStarted","Data":"bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2"} Nov 03 22:38:58.145149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:58.145123 4898 status_manager.go:462] "Container readiness changed for unknown container" pod="opendatahub/maas-controller-b656c5c54-p66vm" containerID="cri-o://0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6" Nov 03 22:38:58.145149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:58.145151 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:38:59.152804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:38:59.152759 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:39:05.097696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:05.097663 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:39:05.110206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:05.110186 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:39:05.203024 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:05.202996 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:39:09.154311 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:09.154273 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-272st" Nov 03 22:39:10.161986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:10.161954 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 22:39:19.834324 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:39:19.834283 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:39:31.959002 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:31.958961 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq"] Nov 03 22:39:31.963510 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:31.963479 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.019294 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.019262 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-961054-openshift-default-dockercfg-kjq9m\"" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.057763 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4frc\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-kube-api-access-n4frc\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.057828 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.057850 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33eaa6-3be4-46a5-80df-db33aee29a68-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.057894 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.057947 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.057996 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.058017 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.058042 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.058448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.058107 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.073822 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.073776 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq"] Nov 03 22:39:32.158692 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.158657 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.158937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.158702 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.158937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.158742 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.158937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.158883 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4frc\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-kube-api-access-n4frc\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.158937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.158914 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.158935 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33eaa6-3be4-46a5-80df-db33aee29a68-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159030 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159112 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159142 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159421 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159195 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159421 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159233 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159421 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159353 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159596 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159433 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.159762 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.159739 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33eaa6-3be4-46a5-80df-db33aee29a68-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.161559 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.161537 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.161836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.161819 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.220639 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.220565 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.220899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.220719 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4frc\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-kube-api-access-n4frc\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.279448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.279409 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:32.713820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.713765 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq"] Nov 03 22:39:32.714497 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:39:32.714468 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe33eaa6_3be4_46a5_80df_db33aee29a68.slice/crio-fc36c8c254c98e8f719ee912ba16469a0c08681e39b1a66bc19e0841eb96539e WatchSource:0}: Error finding container fc36c8c254c98e8f719ee912ba16469a0c08681e39b1a66bc19e0841eb96539e: Status 404 returned error can't find the container with id fc36c8c254c98e8f719ee912ba16469a0c08681e39b1a66bc19e0841eb96539e Nov 03 22:39:32.716957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.716925 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:32.717033 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.716996 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:32.717070 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:32.717039 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:33.376114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:33.376073 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" event={"ID":"fe33eaa6-3be4-46a5-80df-db33aee29a68","Type":"ContainerStarted","Data":"97998fdebcc28f21cdece0c060b343a287789c0cc75be0e5c824f203d1ac96af"} Nov 03 22:39:33.376765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:33.376170 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" event={"ID":"fe33eaa6-3be4-46a5-80df-db33aee29a68","Type":"ContainerStarted","Data":"fc36c8c254c98e8f719ee912ba16469a0c08681e39b1a66bc19e0841eb96539e"} Nov 03 22:39:33.499633 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:33.499580 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podStartSLOduration=2.49956538 podStartE2EDuration="2.49956538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:33.497201636 +0000 UTC m=+1016.709970739" watchObservedRunningTime="2026-11-03 22:39:33.49956538 +0000 UTC m=+1016.712334430" Nov 03 22:39:34.280267 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:34.280226 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:34.281546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:34.281516 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:39:34.281678 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:34.281571 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 22:39:35.280083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:35.280036 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:39:35.280478 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:35.280107 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 22:39:36.280339 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:36.280292 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:39:36.280769 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:36.280364 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 22:39:37.280310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:37.280270 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:39:37.280482 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:37.280333 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 22:39:38.280728 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.280694 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:39:38.281133 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.280748 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 22:39:38.418485 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.418453 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6"] Nov 03 22:39:38.422572 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.422556 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.428706 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.428674 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvp4r\" (UniqueName: \"kubernetes.io/projected/95fd0e1c-3bb7-4404-be60-6992b66de9ee-kube-api-access-nvp4r\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.428856 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.428727 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tmp-dir\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.428906 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.428851 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tls-certs\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.428906 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.428899 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-dshm\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.428986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.428915 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-home\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.429064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.429044 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-model-cache\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.475666 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.475632 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-545b5d\"/\"openshift-service-ca.crt\"" Nov 03 22:39:38.478425 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.478396 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6"] Nov 03 22:39:38.525901 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.525856 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-545b5d\"/\"kube-root-ca.crt\"" Nov 03 22:39:38.526054 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.525923 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-545b5d\"/\"default-dockercfg-s4kxl\"" Nov 03 22:39:38.526054 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.525999 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-545b5d\"/\"distinct-w1-545b5d-kserve-self-signed-certs\"" Nov 03 22:39:38.529643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.529617 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvp4r\" (UniqueName: \"kubernetes.io/projected/95fd0e1c-3bb7-4404-be60-6992b66de9ee-kube-api-access-nvp4r\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.529797 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.529656 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tmp-dir\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.529797 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.529704 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tls-certs\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.529797 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.529746 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-dshm\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.529797 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.529763 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-home\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.530018 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.529826 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-model-cache\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.530168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.530150 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tmp-dir\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.530244 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.530188 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-model-cache\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.530300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.530276 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-home\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.532123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.532071 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-dshm\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.532372 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.532352 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tls-certs\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.592083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.592051 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvp4r\" (UniqueName: \"kubernetes.io/projected/95fd0e1c-3bb7-4404-be60-6992b66de9ee-kube-api-access-nvp4r\") pod \"distinct-w1-545b5d-kserve-5957478b5f-7rtv6\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:38.755006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:38.754959 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:39.178418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:39.178388 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6"] Nov 03 22:39:39.180252 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:39:39.180222 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95fd0e1c_3bb7_4404_be60_6992b66de9ee.slice/crio-34f59e874c9181d21d1b4dcd630ae1590c5aaa5ea5f7a5a705201150e24999fe WatchSource:0}: Error finding container 34f59e874c9181d21d1b4dcd630ae1590c5aaa5ea5f7a5a705201150e24999fe: Status 404 returned error can't find the container with id 34f59e874c9181d21d1b4dcd630ae1590c5aaa5ea5f7a5a705201150e24999fe Nov 03 22:39:39.280728 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:39.280697 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:39:39.280873 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:39.280756 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 22:39:39.416682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:39.416647 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" event={"ID":"95fd0e1c-3bb7-4404-be60-6992b66de9ee","Type":"ContainerStarted","Data":"7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453"} Nov 03 22:39:39.416870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:39.416695 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" event={"ID":"95fd0e1c-3bb7-4404-be60-6992b66de9ee","Type":"ContainerStarted","Data":"34f59e874c9181d21d1b4dcd630ae1590c5aaa5ea5f7a5a705201150e24999fe"} Nov 03 22:39:39.536141 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:39.536021 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podStartSLOduration=1.536006334 podStartE2EDuration="1.536006334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:39.533613455 +0000 UTC m=+1022.746382519" watchObservedRunningTime="2026-11-03 22:39:39.536006334 +0000 UTC m=+1022.748775377" Nov 03 22:39:39.755724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:39.755691 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:40.284914 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:40.284883 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:40.285314 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:40.285129 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:40.285812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:40.285773 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 22:39:48.756670 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:48.756574 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:48.770539 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:48.770511 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:49.501801 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:49.501756 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:39:49.835916 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:39:49.835880 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:39:56.746458 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.746422 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5"] Nov 03 22:39:56.750719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.750698 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.803752 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.803725 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18c03a-openshift-default-dockercfg-p87mq\"" Nov 03 22:39:56.840876 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.840839 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.840876 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.840877 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.841116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.840903 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9bp\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-kube-api-access-zb9bp\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.841116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.841032 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.841116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.841090 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.841231 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.841128 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.841262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.841220 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.841296 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.841277 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.841396 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.841379 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.859286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.859253 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5"] Nov 03 22:39:56.942352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942324 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942372 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942486 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942604 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942522 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942680 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942658 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942737 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942704 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942834 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942841 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.942977 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942958 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.943031 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.942985 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.943031 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.943017 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9bp\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-kube-api-access-zb9bp\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.943135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.943032 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.943135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.943115 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.943302 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.943284 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.945375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.945352 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:56.945495 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:56.945479 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:57.004274 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.004210 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9bp\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-kube-api-access-zb9bp\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:57.004463 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.004443 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:57.064830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.064796 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:57.499510 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.499482 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5"] Nov 03 22:39:57.502310 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:39:57.502278 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9435ac03_5c16_4fe5_ba56_a5934c4f5a8e.slice/crio-d08451f6f267b9c6bf1050cb527e8fc6f6bd134b0febf98d30a86cc679255fd1 WatchSource:0}: Error finding container d08451f6f267b9c6bf1050cb527e8fc6f6bd134b0febf98d30a86cc679255fd1: Status 404 returned error can't find the container with id d08451f6f267b9c6bf1050cb527e8fc6f6bd134b0febf98d30a86cc679255fd1 Nov 03 22:39:57.505222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.505188 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:57.505316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.505259 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:57.505316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.505302 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:57.567394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:57.567367 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" event={"ID":"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e","Type":"ContainerStarted","Data":"d08451f6f267b9c6bf1050cb527e8fc6f6bd134b0febf98d30a86cc679255fd1"} Nov 03 22:39:58.574731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:58.574691 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" event={"ID":"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e","Type":"ContainerStarted","Data":"c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc"} Nov 03 22:39:58.747805 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:58.747724 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podStartSLOduration=2.747691732 podStartE2EDuration="2.747691732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:58.746883918 +0000 UTC m=+1041.959652970" watchObservedRunningTime="2026-11-03 22:39:58.747691732 +0000 UTC m=+1041.960460785" Nov 03 22:39:59.065869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:59.065096 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:39:59.067154 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:59.067118 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:39:59.067279 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:39:59.067178 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 22:40:00.065273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:00.065231 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:40:00.065834 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:00.065296 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 22:40:01.068303 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:01.068259 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:40:01.068984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:01.068868 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 22:40:02.065309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:02.065267 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:40:02.065609 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:02.065576 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 22:40:03.065455 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:03.065118 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:40:03.065455 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:03.065170 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 22:40:04.065421 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:04.065376 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:40:04.065623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:04.065446 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 22:40:05.070104 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:05.070064 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:40:05.070587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:05.070365 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:40:05.071206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:05.071184 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:40:18.009255 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.009225 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg"] Nov 03 22:40:18.014243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.014226 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090148 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090114 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090148 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090156 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090191 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b96c70-3f89-4389-8e41-8f79ff7e3493-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090312 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090382 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090515 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090458 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090515 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090505 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090597 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090536 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.090659 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.090643 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzpv\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-kube-api-access-6kzpv\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.119930 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.119891 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg"] Nov 03 22:40:18.191956 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.191922 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzpv\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-kube-api-access-6kzpv\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.191975 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.191996 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192032 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b96c70-3f89-4389-8e41-8f79ff7e3493-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192065 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192111 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192179 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192218 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192247 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192430 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192627 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192468 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.192930 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.192897 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b96c70-3f89-4389-8e41-8f79ff7e3493-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.193112 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.193086 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.193194 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.193146 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.194677 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.194648 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.194829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.194782 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.256162 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.256126 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzpv\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-kube-api-access-6kzpv\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.257567 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.257539 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.328674 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.328638 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:18.762953 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:40:18.762923 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b96c70_3f89_4389_8e41_8f79ff7e3493.slice/crio-6b49f9ee173c1ec3be07cdb5a771f2c86a10ca99e9817642be75dbda6f01fd20 WatchSource:0}: Error finding container 6b49f9ee173c1ec3be07cdb5a771f2c86a10ca99e9817642be75dbda6f01fd20: Status 404 returned error can't find the container with id 6b49f9ee173c1ec3be07cdb5a771f2c86a10ca99e9817642be75dbda6f01fd20 Nov 03 22:40:18.763093 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.763027 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg"] Nov 03 22:40:18.765861 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.765820 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:40:18.765954 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.765914 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:40:18.765995 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:18.765962 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:40:19.699699 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:19.699663 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" event={"ID":"62b96c70-3f89-4389-8e41-8f79ff7e3493","Type":"ContainerStarted","Data":"35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c"} Nov 03 22:40:19.700227 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:19.699725 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" event={"ID":"62b96c70-3f89-4389-8e41-8f79ff7e3493","Type":"ContainerStarted","Data":"6b49f9ee173c1ec3be07cdb5a771f2c86a10ca99e9817642be75dbda6f01fd20"} Nov 03 22:40:19.862288 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:40:19.862179 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:40:19.878372 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:19.878319 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podStartSLOduration=2.878306356 podStartE2EDuration="2.878306356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:19.877200254 +0000 UTC m=+1063.089969304" watchObservedRunningTime="2026-11-03 22:40:19.878306356 +0000 UTC m=+1063.091075406" Nov 03 22:40:20.329523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:20.329488 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:20.331007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:20.330980 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:40:20.331131 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:20.331031 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:40:21.329097 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:21.329059 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:40:21.329497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:21.329124 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:40:22.329897 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:22.329863 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:40:22.330264 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:22.329917 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:40:23.329565 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:23.329519 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:40:23.329753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:23.329601 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:40:24.329809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:24.329760 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:40:24.330223 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:24.329838 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:40:25.329750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:25.329714 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:40:25.329992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:25.329871 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:40:26.333937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:26.333905 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:26.334366 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:26.334173 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:26.335108 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:40:26.335086 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:40:49.834813 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:40:49.834740 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:41:17.784867 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.784769 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6"] Nov 03 22:41:17.788885 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.788867 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.894633 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.894601 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6"] Nov 03 22:41:17.924497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924467 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.924652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924578 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.924652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924602 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplfm\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-kube-api-access-xplfm\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.924652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924628 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.924773 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924758 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.924835 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924815 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.924938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924878 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35089b8e-3361-4bbb-844b-7284485d726f-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.924938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924917 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:17.925013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:17.924957 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35089b8e-3361-4bbb-844b-7284485d726f-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026097 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026055 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026171 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026199 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplfm\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-kube-api-access-xplfm\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026236 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026374 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026403 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026452 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35089b8e-3361-4bbb-844b-7284485d726f-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026632 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026489 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026632 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026524 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35089b8e-3361-4bbb-844b-7284485d726f-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026632 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026606 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026780 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026663 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026780 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026716 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.026938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.026920 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.027214 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.027189 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35089b8e-3361-4bbb-844b-7284485d726f-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.028885 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.028854 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.029149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.029132 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35089b8e-3361-4bbb-844b-7284485d726f-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.087298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.087268 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.087481 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.087403 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplfm\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-kube-api-access-xplfm\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.102277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.102246 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:18.534330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.534298 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6"] Nov 03 22:41:18.536608 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:41:18.536581 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35089b8e_3361_4bbb_844b_7284485d726f.slice/crio-3cd23ce52ba1d099c758bde5a0431b53576da5ad41afba5ec9d08f7435de1919 WatchSource:0}: Error finding container 3cd23ce52ba1d099c758bde5a0431b53576da5ad41afba5ec9d08f7435de1919: Status 404 returned error can't find the container with id 3cd23ce52ba1d099c758bde5a0431b53576da5ad41afba5ec9d08f7435de1919 Nov 03 22:41:18.539591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.539552 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:18.539694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.539623 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:18.539694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.539655 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:18.792609 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.792513 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh"] Nov 03 22:41:18.797520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.797494 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.852123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.852092 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-68fbd1-openshift-default-dockercfg-sd2d7\"" Nov 03 22:41:18.856444 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.856361 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh"] Nov 03 22:41:18.938008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.937977 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938055 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938145 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938279 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938258 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48419746-5dd4-4d86-8382-84ec8283fe40-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938327 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938311 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48419746-5dd4-4d86-8382-84ec8283fe40-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938426 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938408 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zg5\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-kube-api-access-n5zg5\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938495 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938450 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938563 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938545 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:18.938638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:18.938621 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039361 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039331 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48419746-5dd4-4d86-8382-84ec8283fe40-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039384 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48419746-5dd4-4d86-8382-84ec8283fe40-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039418 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zg5\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-kube-api-access-n5zg5\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039452 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039493 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039581 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039627 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039798 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039761 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.039893 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.039816 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.040504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.040253 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.040504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.040299 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.040678 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.040616 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.040678 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.040618 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.040972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.040950 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48419746-5dd4-4d86-8382-84ec8283fe40-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.042282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.042256 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.043233 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.043094 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48419746-5dd4-4d86-8382-84ec8283fe40-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.079111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.079067 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" event={"ID":"35089b8e-3361-4bbb-844b-7284485d726f","Type":"ContainerStarted","Data":"a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d"} Nov 03 22:41:19.079299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.079141 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" event={"ID":"35089b8e-3361-4bbb-844b-7284485d726f","Type":"ContainerStarted","Data":"3cd23ce52ba1d099c758bde5a0431b53576da5ad41afba5ec9d08f7435de1919"} Nov 03 22:41:19.108812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.105567 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.108812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.106087 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:19.108812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.106306 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:41:19.108812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.106355 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 22:41:19.108812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.108686 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zg5\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-kube-api-access-n5zg5\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.114940 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.114912 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:19.218520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.217967 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podStartSLOduration=2.217942486 podStartE2EDuration="2.217942486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:19.202685366 +0000 UTC m=+1122.415454420" watchObservedRunningTime="2026-11-03 22:41:19.217942486 +0000 UTC m=+1122.430711538" Nov 03 22:41:19.580006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.579974 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh"] Nov 03 22:41:19.581899 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:41:19.581863 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48419746_5dd4_4d86_8382_84ec8283fe40.slice/crio-fee1480320695e44eacac23c33a926e6d1fe8b2bf874ab4cc3e839d32eba0ca0 WatchSource:0}: Error finding container fee1480320695e44eacac23c33a926e6d1fe8b2bf874ab4cc3e839d32eba0ca0: Status 404 returned error can't find the container with id fee1480320695e44eacac23c33a926e6d1fe8b2bf874ab4cc3e839d32eba0ca0 Nov 03 22:41:19.584507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.584470 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:19.584617 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.584553 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:19.584617 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:19.584598 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:19.874669 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:41:19.874584 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:41:20.091085 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:20.091043 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" event={"ID":"48419746-5dd4-4d86-8382-84ec8283fe40","Type":"ContainerStarted","Data":"b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6"} Nov 03 22:41:20.091254 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:20.091129 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" event={"ID":"48419746-5dd4-4d86-8382-84ec8283fe40","Type":"ContainerStarted","Data":"fee1480320695e44eacac23c33a926e6d1fe8b2bf874ab4cc3e839d32eba0ca0"} Nov 03 22:41:20.102763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:20.102734 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:41:20.102948 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:20.102801 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 22:41:20.115938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:20.115908 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:20.117633 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:20.117601 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:41:20.117746 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:20.117650 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:41:21.103526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:21.103488 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:41:21.104046 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:21.103538 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 22:41:21.115933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:21.115904 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:41:21.116052 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:21.115944 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:41:22.103385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:22.103341 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:41:22.103572 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:22.103414 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 22:41:22.115694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:22.115661 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:41:22.115820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:22.115722 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:41:23.103683 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:23.103637 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:41:23.104150 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:23.103725 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 22:41:23.115440 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:23.115415 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:41:23.115660 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:23.115640 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:41:24.111243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:24.111196 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:41:24.111778 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:24.111261 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 22:41:24.116324 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:24.116293 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:41:24.116569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:24.116522 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:41:25.102986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:25.102945 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:41:25.103169 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:25.103010 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 22:41:25.116107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:25.116069 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:41:25.116573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:25.116114 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:41:26.116063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:26.116026 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:41:26.116351 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:26.116322 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:41:26.280684 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:26.280648 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:26.281590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:26.281561 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:26.282488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:26.282465 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:41:26.456146 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:26.456032 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podStartSLOduration=8.456014843 podStartE2EDuration="8.456014843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:20.222507319 +0000 UTC m=+1123.435276384" watchObservedRunningTime="2026-11-03 22:41:26.456014843 +0000 UTC m=+1129.668783895" Nov 03 22:41:28.156988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:28.156954 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:41:28.157560 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:28.157531 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:28.477011 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:28.476927 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:28.477248 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:28.477229 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:28.477948 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:28.477932 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:41:49.334803 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.334763 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9"] Nov 03 22:41:49.340021 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.339998 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.403541 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403510 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.403721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403560 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.403721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403659 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.403933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403719 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.403933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403749 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rhr\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-kube-api-access-n8rhr\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.403933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403867 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.404072 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403939 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.404072 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.403990 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.404147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.404073 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.456164 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.456129 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9"] Nov 03 22:41:49.506066 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506026 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506248 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506134 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506248 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506189 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506367 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506320 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506367 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506357 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rhr\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-kube-api-access-n8rhr\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506477 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506394 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506477 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506458 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506589 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506513 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506589 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506513 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506589 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506571 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506601 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506611 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.506872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.506838 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.507434 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.507407 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.509092 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.509064 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.510089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.510064 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.568534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.568500 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.568736 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.568711 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rhr\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-kube-api-access-n8rhr\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.654629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:49.654551 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:49.836267 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:41:49.836231 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:41:50.085824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.085394 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9"] Nov 03 22:41:50.090395 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:41:50.090367 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5057cf70_852c_412d_bf26_dbf1f53c4c7b.slice/crio-0c49c349b3a51613d730075605c15ffaf083fa04099b6a4612971277aa245bb0 WatchSource:0}: Error finding container 0c49c349b3a51613d730075605c15ffaf083fa04099b6a4612971277aa245bb0: Status 404 returned error can't find the container with id 0c49c349b3a51613d730075605c15ffaf083fa04099b6a4612971277aa245bb0 Nov 03 22:41:50.092866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.092832 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:50.092946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.092903 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:50.092946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.092934 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:50.299582 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.299537 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" event={"ID":"5057cf70-852c-412d-bf26-dbf1f53c4c7b","Type":"ContainerStarted","Data":"efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69"} Nov 03 22:41:50.299746 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.299609 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" event={"ID":"5057cf70-852c-412d-bf26-dbf1f53c4c7b","Type":"ContainerStarted","Data":"0c49c349b3a51613d730075605c15ffaf083fa04099b6a4612971277aa245bb0"} Nov 03 22:41:50.423350 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.423246 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podStartSLOduration=1.42323084 podStartE2EDuration="1.42323084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:50.42017956 +0000 UTC m=+1153.632948634" watchObservedRunningTime="2026-11-03 22:41:50.42323084 +0000 UTC m=+1153.635999891" Nov 03 22:41:50.658813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:50.657868 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:51.674981 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:51.674943 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:41:51.675661 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:51.675629 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 22:41:52.655374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:52.655333 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:41:52.655711 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:52.655662 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 22:41:53.659847 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:53.659731 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:41:53.660393 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:53.659990 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 22:41:54.655957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:54.655919 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:41:54.656508 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:54.656477 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 22:41:55.655230 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:55.655189 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:41:55.655901 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:55.655857 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 22:41:56.655188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:56.655154 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:41:56.655388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:56.655209 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 22:41:57.659471 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:57.659430 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:57.659939 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:57.659834 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:41:57.660740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:41:57.660717 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:42:02.078007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.077974 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dwnbd-4pz2h"] Nov 03 22:42:02.082317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.082296 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.158899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.158130 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-service-ca-bundle\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.158899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.158277 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-trusted-ca-bundle\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.158899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.158345 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/78156a15-e7f0-4de1-a254-20e2e4a16100-archives-path\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.158899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.158419 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9bpd\" (UniqueName: \"kubernetes.io/projected/78156a15-e7f0-4de1-a254-20e2e4a16100-kube-api-access-n9bpd\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.160421 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.160384 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dwnbd-4pz2h"] Nov 03 22:42:02.259123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.259090 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-trusted-ca-bundle\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.259330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.259148 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/78156a15-e7f0-4de1-a254-20e2e4a16100-archives-path\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.259330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.259201 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9bpd\" (UniqueName: \"kubernetes.io/projected/78156a15-e7f0-4de1-a254-20e2e4a16100-kube-api-access-n9bpd\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.259469 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.259362 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-service-ca-bundle\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.259643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.259624 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/78156a15-e7f0-4de1-a254-20e2e4a16100-archives-path\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.260037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.260013 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-trusted-ca-bundle\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.260136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.260088 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-service-ca-bundle\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.321231 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.321198 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9bpd\" (UniqueName: \"kubernetes.io/projected/78156a15-e7f0-4de1-a254-20e2e4a16100-kube-api-access-n9bpd\") pod \"periodic-gathering-dwnbd-4pz2h\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.396259 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.396176 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:42:02.781186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:02.780319 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dwnbd-4pz2h"] Nov 03 22:42:02.783647 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:42:02.783619 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78156a15_e7f0_4de1_a254_20e2e4a16100.slice/crio-dfa6f8398ef290ae31f680182c67a6af90c92901ddbcc98c423fc85594bc751b WatchSource:0}: Error finding container dfa6f8398ef290ae31f680182c67a6af90c92901ddbcc98c423fc85594bc751b: Status 404 returned error can't find the container with id dfa6f8398ef290ae31f680182c67a6af90c92901ddbcc98c423fc85594bc751b Nov 03 22:42:03.410644 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:03.410599 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" event={"ID":"78156a15-e7f0-4de1-a254-20e2e4a16100","Type":"ContainerStarted","Data":"905ac8fc9e4bf28cfaa0b06faae8931fdc29f71797682c08e257daa28081b171"} Nov 03 22:42:03.411070 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:03.410660 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" event={"ID":"78156a15-e7f0-4de1-a254-20e2e4a16100","Type":"ContainerStarted","Data":"dfa6f8398ef290ae31f680182c67a6af90c92901ddbcc98c423fc85594bc751b"} Nov 03 22:42:03.528533 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:03.528478 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" podStartSLOduration=1.528462716 podStartE2EDuration="1.528462716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:03.526915153 +0000 UTC m=+1166.739684227" watchObservedRunningTime="2026-11-03 22:42:03.528462716 +0000 UTC m=+1166.741231768" Nov 03 22:42:04.446803 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.446755 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629"] Nov 03 22:42:04.451127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.451107 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.548848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.548810 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629"] Nov 03 22:42:04.590669 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.590624 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.590669 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.590673 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.590930 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.590833 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.590930 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.590886 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.591006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.590946 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.591103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.591083 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fpl\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-kube-api-access-24fpl\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.591138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.591127 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.591311 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.591290 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636b01e1-8ade-44d2-af06-5c85804832e7-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.591359 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.591343 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636b01e1-8ade-44d2-af06-5c85804832e7-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.692615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.692578 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636b01e1-8ade-44d2-af06-5c85804832e7-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.692822 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.692632 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636b01e1-8ade-44d2-af06-5c85804832e7-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.692903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.692823 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.692903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.692846 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.692903 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.692952 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693074 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693225 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693200 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fpl\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-kube-api-access-24fpl\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693252 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693259 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693276 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693398 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693584 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636b01e1-8ade-44d2-af06-5c85804832e7-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.693700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.693676 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.695296 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.695270 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636b01e1-8ade-44d2-af06-5c85804832e7-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.695385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.695371 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.757268 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.757193 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.757268 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.757252 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fpl\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-kube-api-access-24fpl\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:04.765048 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:04.765026 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:05.196850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.196782 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629"] Nov 03 22:42:05.197949 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:42:05.197921 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636b01e1_8ade_44d2_af06_5c85804832e7.slice/crio-fe4a1c5504cb000659e2794f4e999bc0017be5a8392852930837a59b5538a0c5 WatchSource:0}: Error finding container fe4a1c5504cb000659e2794f4e999bc0017be5a8392852930837a59b5538a0c5: Status 404 returned error can't find the container with id fe4a1c5504cb000659e2794f4e999bc0017be5a8392852930837a59b5538a0c5 Nov 03 22:42:05.200453 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.200422 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:05.200537 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.200487 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:05.200537 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.200516 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:05.425077 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.425026 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" event={"ID":"636b01e1-8ade-44d2-af06-5c85804832e7","Type":"ContainerStarted","Data":"5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8"} Nov 03 22:42:05.425251 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.425097 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" event={"ID":"636b01e1-8ade-44d2-af06-5c85804832e7","Type":"ContainerStarted","Data":"fe4a1c5504cb000659e2794f4e999bc0017be5a8392852930837a59b5538a0c5"} Nov 03 22:42:05.547836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.547714 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podStartSLOduration=1.5476990339999999 podStartE2EDuration="1.547699034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:05.544930331 +0000 UTC m=+1168.757699402" watchObservedRunningTime="2026-11-03 22:42:05.547699034 +0000 UTC m=+1168.760468148" Nov 03 22:42:05.765968 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:05.765927 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:06.766628 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:06.766595 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:42:06.767051 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:06.766646 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 22:42:07.766071 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:07.766028 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:42:07.766289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:07.766097 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 22:42:08.766387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:08.766348 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:42:08.766830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:08.766417 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 22:42:09.766321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:09.766281 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:42:09.766521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:09.766350 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 22:42:10.766373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:10.766338 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:42:10.766549 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:10.766395 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 22:42:11.765666 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:11.765628 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:42:11.765873 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:11.765693 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 22:42:12.745188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.745147 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n"] Nov 03 22:42:12.749598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.749573 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.771089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.771047 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:12.771551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.771403 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:12.772275 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.772256 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:42:12.842130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.842092 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n"] Nov 03 22:42:12.923593 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.923562 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.923825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.923783 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.923998 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.923841 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.923998 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.923872 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00993844-beef-4932-b545-d4f3dc410c15-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.924176 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.924150 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnf86\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-kube-api-access-fnf86\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.924431 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.924411 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.925023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.924697 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00993844-beef-4932-b545-d4f3dc410c15-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.925023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.924856 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:12.925023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:12.924892 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.025868 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.025767 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.025868 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.025829 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.025868 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.025850 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00993844-beef-4932-b545-d4f3dc410c15-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.025882 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnf86\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-kube-api-access-fnf86\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.025920 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.025954 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00993844-beef-4932-b545-d4f3dc410c15-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.026003 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.026020 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.026053 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026455 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.026187 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026455 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.026198 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.026567 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.026880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.026856 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00993844-beef-4932-b545-d4f3dc410c15-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.027102 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.027080 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.028509 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.028484 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.029147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.029124 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00993844-beef-4932-b545-d4f3dc410c15-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.135568 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.135501 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnf86\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-kube-api-access-fnf86\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.136297 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.136272 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.367931 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.367888 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:13.806663 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.806635 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n"] Nov 03 22:42:13.808022 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:42:13.807994 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00993844_beef_4932_b545_d4f3dc410c15.slice/crio-03eaf89d3be988be359dcdf92b28119ac467fc9861bc629c79aeb563838a4b8b WatchSource:0}: Error finding container 03eaf89d3be988be359dcdf92b28119ac467fc9861bc629c79aeb563838a4b8b: Status 404 returned error can't find the container with id 03eaf89d3be988be359dcdf92b28119ac467fc9861bc629c79aeb563838a4b8b Nov 03 22:42:13.810912 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.810874 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:13.810977 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.810950 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:13.811021 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:13.810995 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:14.516576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:14.516540 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" event={"ID":"00993844-beef-4932-b545-d4f3dc410c15","Type":"ContainerStarted","Data":"ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c"} Nov 03 22:42:14.516756 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:14.516608 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" event={"ID":"00993844-beef-4932-b545-d4f3dc410c15","Type":"ContainerStarted","Data":"03eaf89d3be988be359dcdf92b28119ac467fc9861bc629c79aeb563838a4b8b"} Nov 03 22:42:14.692000 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:14.691944 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podStartSLOduration=2.691929929 podStartE2EDuration="2.691929929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:14.689139032 +0000 UTC m=+1177.901908084" watchObservedRunningTime="2026-11-03 22:42:14.691929929 +0000 UTC m=+1177.904698979" Nov 03 22:42:15.368270 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:15.368234 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:15.370098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:15.370072 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:42:15.370212 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:15.370118 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:42:16.379600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:16.378605 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:42:16.379600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:16.378666 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:42:17.372302 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:17.372265 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:42:17.372490 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:17.372324 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:42:18.306379 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.306337 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz"] Nov 03 22:42:18.313260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.313233 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.369271 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.369230 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:42:18.369449 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.369299 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:42:18.401533 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.401499 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aff2e6-4643-40f2-a714-46ec991854a4-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.401828 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.401781 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.401994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.401977 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.402197 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.402179 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aff2e6-4643-40f2-a714-46ec991854a4-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.402358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.402328 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvgg6\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-kube-api-access-qvgg6\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.402433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.402388 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.402433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.402415 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.402544 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.402444 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.402544 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.402495 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.471208 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.471171 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz"] Nov 03 22:42:18.503972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.503937 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aff2e6-4643-40f2-a714-46ec991854a4-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504022 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504078 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504109 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aff2e6-4643-40f2-a714-46ec991854a4-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504164 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvgg6\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-kube-api-access-qvgg6\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504396 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504207 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504396 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504235 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504396 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504266 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.504396 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504315 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.505028 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.504996 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.505285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.505261 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.505894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.505414 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aff2e6-4643-40f2-a714-46ec991854a4-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.505980 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.505905 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.506082 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.506060 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.509277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.509245 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.511576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.511560 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aff2e6-4643-40f2-a714-46ec991854a4-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.570045 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.570014 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd"] Nov 03 22:42:18.573391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.573358 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvgg6\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-kube-api-access-qvgg6\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.575338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.575282 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.577018 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.577002 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.605682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.605650 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.605876 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.605772 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662d4\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-kube-api-access-662d4\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.605876 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.605869 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.605954 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.605932 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.605989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.605970 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.606135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.606110 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.606170 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.606155 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.606234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.606212 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.606283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.606240 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.629542 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.629514 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd"] Nov 03 22:42:18.630677 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.630651 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5e72ad-openshift-default-dockercfg-5qnwn\"" Nov 03 22:42:18.631763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.631740 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:18.708276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708239 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662d4\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-kube-api-access-662d4\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708328 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708464 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708402 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708464 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708442 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708554 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708620 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708610 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708658 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708718 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708701 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.708799 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.708769 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.709384 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.709356 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.710072 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.710043 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.710325 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.710304 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.710623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.710598 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.711080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.711054 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.713599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.713570 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.713825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.713802 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.772310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.772201 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662d4\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-kube-api-access-662d4\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.772310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.772215 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:18.915036 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:18.914952 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:19.162407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.162372 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz"] Nov 03 22:42:19.162905 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:42:19.162864 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aff2e6_4643_40f2_a714_46ec991854a4.slice/crio-c068fec52269b5b0d132a944dfb4aa3f0ce5d276afc3f34b6d58c0137874249d WatchSource:0}: Error finding container c068fec52269b5b0d132a944dfb4aa3f0ce5d276afc3f34b6d58c0137874249d: Status 404 returned error can't find the container with id c068fec52269b5b0d132a944dfb4aa3f0ce5d276afc3f34b6d58c0137874249d Nov 03 22:42:19.165739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.165666 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.165739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.165735 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.165939 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.165767 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.261331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.261301 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst"] Nov 03 22:42:19.265716 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.265697 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.321807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.321002 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst"] Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.322914 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcmjb\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-kube-api-access-lcmjb\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323072 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323157 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323321 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323354 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ee53f-509e-429a-b430-23f828e0cfb0-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323527 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323600 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323626 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.329945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.323775 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.387764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.356341 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd"] Nov 03 22:42:19.387764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.377521 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:42:19.387764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.377589 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:42:19.395637 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.395594 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.395723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.395689 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.395798 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.395747 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.424672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424646 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.424801 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424683 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ee53f-509e-429a-b430-23f828e0cfb0-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.424801 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424729 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.424801 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424753 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.424801 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424769 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.425013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424845 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.425013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424894 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcmjb\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-kube-api-access-lcmjb\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.425013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424918 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.425013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.424965 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.425830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.425462 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.425830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.425756 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.425987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.425848 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.426162 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.426120 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.426225 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.426199 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ee53f-509e-429a-b430-23f828e0cfb0-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.426992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.426973 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.427592 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.427574 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.488369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.486540 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.488369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.487331 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcmjb\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-kube-api-access-lcmjb\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.587990 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.587958 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:19.600410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.600370 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" event={"ID":"95aff2e6-4643-40f2-a714-46ec991854a4","Type":"ContainerStarted","Data":"be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02"} Nov 03 22:42:19.601016 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.600987 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" event={"ID":"95aff2e6-4643-40f2-a714-46ec991854a4","Type":"ContainerStarted","Data":"c068fec52269b5b0d132a944dfb4aa3f0ce5d276afc3f34b6d58c0137874249d"} Nov 03 22:42:19.602218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.602192 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" event={"ID":"b86c6c87-ccf4-433e-bf67-850af6e8ec57","Type":"ContainerStarted","Data":"be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c"} Nov 03 22:42:19.602303 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.602238 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" event={"ID":"b86c6c87-ccf4-433e-bf67-850af6e8ec57","Type":"ContainerStarted","Data":"aff0d3169f6622e28aa5bf45dd024f01ed2ab47b1f143e6b566363dc127c6021"} Nov 03 22:42:19.632040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.631997 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:19.758168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.758035 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podStartSLOduration=1.7580167169999998 podStartE2EDuration="1.758016717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:19.752173072 +0000 UTC m=+1182.964942154" watchObservedRunningTime="2026-11-03 22:42:19.758016717 +0000 UTC m=+1182.970785770" Nov 03 22:42:19.881745 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.881672 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podStartSLOduration=1.881628385 podStartE2EDuration="1.881628385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:19.879687647 +0000 UTC m=+1183.092456756" watchObservedRunningTime="2026-11-03 22:42:19.881628385 +0000 UTC m=+1183.094397437" Nov 03 22:42:19.887390 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:42:19.887131 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:42:19.915975 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:19.915927 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:20.034866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.034062 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst"] Nov 03 22:42:20.068685 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:42:20.068554 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8ee53f_509e_429a_b430_23f828e0cfb0.slice/crio-d8c35df59f715c7342c26110b8bbd896a97517e0d77666521c64058730692416 WatchSource:0}: Error finding container d8c35df59f715c7342c26110b8bbd896a97517e0d77666521c64058730692416: Status 404 returned error can't find the container with id d8c35df59f715c7342c26110b8bbd896a97517e0d77666521c64058730692416 Nov 03 22:42:20.075516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.075471 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:20.075716 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.075696 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:20.075881 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.075862 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:20.373879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.373428 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.373879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.373489 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:42:20.626481 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.626391 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" event={"ID":"1c8ee53f-509e-429a-b430-23f828e0cfb0","Type":"ContainerStarted","Data":"e739971d611b2f74443118aa2c93be71cb1b5ce2dc6fbe73e050c72cebb49f95"} Nov 03 22:42:20.627492 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.627467 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" event={"ID":"1c8ee53f-509e-429a-b430-23f828e0cfb0","Type":"ContainerStarted","Data":"d8c35df59f715c7342c26110b8bbd896a97517e0d77666521c64058730692416"} Nov 03 22:42:20.641935 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.641902 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.642173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.642145 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:42:20.772242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.772172 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podStartSLOduration=1.7721492140000001 podStartE2EDuration="1.772149214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:20.76709205 +0000 UTC m=+1183.979861105" watchObservedRunningTime="2026-11-03 22:42:20.772149214 +0000 UTC m=+1183.984918270" Nov 03 22:42:20.931434 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.931124 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.931434 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:20.931172 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 22:42:21.590894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.590854 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:21.591503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.591064 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.591639 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.591618 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:21.669812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.658325 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.670100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.670064 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:42:21.950921 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.945920 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.950921 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.945986 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 22:42:21.969811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.969091 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:21.970289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.970263 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:21.974068 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:21.974042 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:42:22.595960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:22.595919 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.596451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:22.595985 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:22.715813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:22.708327 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.715813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:22.708393 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:42:22.952270 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:22.952170 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.952547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:22.952508 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 22:42:23.225537 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:23.224899 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:23.588638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:23.588598 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.588949 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:23.588918 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:23.633835 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:23.632216 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.633835 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:23.632285 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:42:23.970872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:23.967087 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.970872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:23.967152 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 22:42:24.590427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:24.590370 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.590769 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:24.590717 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:24.633350 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:24.633306 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.633685 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:24.633654 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:42:24.922065 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:24.921968 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.922625 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:24.922594 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 22:42:25.589128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:25.588656 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:25.589128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:25.588720 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:25.639822 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:25.633864 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:42:25.639822 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:25.633984 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:42:25.919345 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:25.919249 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:42:25.919647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:25.919612 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 22:42:26.589457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:26.589415 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:26.590104 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:26.590066 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:26.632304 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:26.632268 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:42:26.632497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:26.632332 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:42:26.916384 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:26.916301 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:42:26.916384 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:26.916351 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 22:42:27.589217 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:27.589178 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:27.589527 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:27.589480 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:27.924420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:27.924339 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:27.926042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:27.926022 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:27.927400 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:27.927382 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:42:28.140564 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:28.140482 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:28.143214 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:28.141720 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:28.143623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:28.143603 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 22:42:28.588762 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:28.588730 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.588994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:28.588837 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 22:42:29.592919 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:29.592888 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:29.593390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:29.593188 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:29.594026 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:29.594007 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:42:33.768863 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.743414 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj"] Nov 03 22:42:33.812933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.812900 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj"] Nov 03 22:42:33.813340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.813320 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.973831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.973762 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.974404 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.974376 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.974656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.974633 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492n6\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-kube-api-access-492n6\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.974824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.974781 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.974965 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.974948 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.975229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.975190 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.975393 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.975375 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.975573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.975556 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:33.975697 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:33.975680 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30088937-2aae-4df7-b461-bdb0889bf8a1-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.092448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.092408 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.092821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.092777 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.093048 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.093030 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.093174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.093159 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30088937-2aae-4df7-b461-bdb0889bf8a1-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.093324 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.093309 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.093437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.093421 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.093637 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.093618 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492n6\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-kube-api-access-492n6\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.093754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.093740 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.093931 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.093913 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.094594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.094574 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.095182 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.095159 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.095336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.095232 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30088937-2aae-4df7-b461-bdb0889bf8a1-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.095517 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.095492 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.095646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.095609 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.106474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.106409 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.108266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.108232 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.161926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.161833 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.162579 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.162550 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-492n6\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-kube-api-access-492n6\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.270272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.270235 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:34.796158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.794944 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj"] Nov 03 22:42:34.822266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.821645 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:34.822266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.821719 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:34.822266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:34.821756 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:35.368308 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:35.368267 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" event={"ID":"30088937-2aae-4df7-b461-bdb0889bf8a1","Type":"ContainerStarted","Data":"3dccd7923879a872ef08612faa72c92e284149c08315115a5f770ec6aa9cd5e4"} Nov 03 22:42:35.368564 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:35.368545 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" event={"ID":"30088937-2aae-4df7-b461-bdb0889bf8a1","Type":"ContainerStarted","Data":"cb1cd3a1085b17f68064d4c5d808178c67bd5187014ee802bb488fd0257f5a1b"} Nov 03 22:42:35.547378 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:35.547311 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podStartSLOduration=2.5461183050000002 podStartE2EDuration="2.546118305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:35.541887584 +0000 UTC m=+1198.754656636" watchObservedRunningTime="2026-11-03 22:42:35.546118305 +0000 UTC m=+1198.758887357" Nov 03 22:42:36.275126 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:36.275028 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:36.279027 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:36.278980 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:42:36.279301 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:36.279255 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:42:37.193092 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:37.193048 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:42:37.196843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:37.196818 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:42:37.200410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:37.200381 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:42:37.203692 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:37.203665 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:42:37.275995 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:37.275956 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:42:37.276595 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:37.276565 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:42:38.271114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:38.271072 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:42:38.271297 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:38.271154 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:42:39.271466 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:39.271425 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:42:39.271898 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:39.271510 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:42:40.271106 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:40.271063 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:42:40.271288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:40.271141 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:42:41.271275 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:41.271224 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:42:41.271762 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:41.271292 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:42:42.533612 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:42.533581 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:42.534078 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:42.534028 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:42.534997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:42:42.534965 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:42:49.835641 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:42:49.835598 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:43:04.238336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.238296 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd"] Nov 03 22:43:04.242970 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.242940 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.300812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.300755 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5kg\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-kube-api-access-gj5kg\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301017 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.300967 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301085 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.301070 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.301112 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233153b9-fe39-4082-87c1-169894343e6b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301201 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.301184 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.301215 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.301251 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233153b9-fe39-4082-87c1-169894343e6b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301346 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.301275 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.301346 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.301293 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.336363 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.336324 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd"] Nov 03 22:43:04.416685 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.416639 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5kg\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-kube-api-access-gj5kg\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.417725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.417697 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.417991 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.417973 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.418102 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.418090 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233153b9-fe39-4082-87c1-169894343e6b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.418340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.418321 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.418502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.418487 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.418761 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.418625 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233153b9-fe39-4082-87c1-169894343e6b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.460305 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.420731 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.460530 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.421026 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.460650 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.421960 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233153b9-fe39-4082-87c1-169894343e6b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.460733 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.451329 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233153b9-fe39-4082-87c1-169894343e6b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.460989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.460965 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.461437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.461418 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.461922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.461903 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.462053 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.461370 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.481726 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.481695 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.485533 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.485491 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5kg\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-kube-api-access-gj5kg\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.504948 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.504876 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:04.559773 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:04.559739 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:05.120957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:05.120822 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd"] Nov 03 22:43:05.133840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:05.133315 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:05.133840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:05.133410 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:05.133840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:05.133452 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:05.660476 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:05.660432 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" event={"ID":"233153b9-fe39-4082-87c1-169894343e6b","Type":"ContainerStarted","Data":"7ac5b606e3efadba1724f0104ea2b432b99fbcdd7f5b9fc053178d0a6f2e914b"} Nov 03 22:43:06.357812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.356992 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv"] Nov 03 22:43:06.357812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.357504 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" containerID="cri-o://a5e02f23aa879339b649c94d1c5c5710536d49a94379c1bf492284b36ce10530" gracePeriod=30 Nov 03 22:43:06.526978 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.526935 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj"] Nov 03 22:43:06.527572 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.527517 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" containerID="cri-o://da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c" gracePeriod=30 Nov 03 22:43:06.651547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.629006 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct"] Nov 03 22:43:06.651547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.629286 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" containerID="cri-o://313b5003e527a56e8d3d9a11a983d3ab8065351324f1e3bda474f803c32f12f6" gracePeriod=30 Nov 03 22:43:06.774855 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.773877 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-d5brv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:43:06.774855 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.773961 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:43:06.828320 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.828279 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" event={"ID":"233153b9-fe39-4082-87c1-169894343e6b","Type":"ContainerStarted","Data":"d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0"} Nov 03 22:43:06.862307 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.861858 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x"] Nov 03 22:43:06.862307 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:06.862103 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" containerID="cri-o://24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690" gracePeriod=30 Nov 03 22:43:07.016869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.012869 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:07.211188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.211137 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:43:07.212370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.212331 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 22:43:07.531204 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.531157 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.557812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.552778 4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.581265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.581233 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:07.582900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.582873 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.583088 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:07.583066 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:08.044815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.036245 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:08.044815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.036320 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.070686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.037165 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.070686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.061342 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:43:08.224329 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.224296 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:43:08.225909 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.224159 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.245513 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.245481 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:43:08.245844 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.245829 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:43:08.457823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.451332 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podStartSLOduration=4.451309282 podStartE2EDuration="4.451309282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:07.743568312 +0000 UTC m=+1230.956337366" watchObservedRunningTime="2026-11-03 22:43:08.451309282 +0000 UTC m=+1231.664078334" Nov 03 22:43:08.534703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.534661 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.536538 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.536512 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:43:08.635495 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.635451 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.635810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:08.635759 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:09.060709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:09.060667 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.238586 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:09.227344 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.267090 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:09.249667 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:43:09.536923 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:09.536876 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.587847 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:09.576632 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:43:09.613837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:09.613778 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:09.614117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:09.614086 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:10.072978 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.072934 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.214516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.212869 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:43:10.250813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.244059 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:10.250813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.244124 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.279302 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.279174 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j72ct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:43:10.279302 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.279233 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:43:10.514845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.514749 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:43:10.578184 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.578150 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:10.578451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:10.578421 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:11.518459 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.518427 4898 generic.go:356] "Generic (PLEG): container finished" podID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerID="a5e02f23aa879339b649c94d1c5c5710536d49a94379c1bf492284b36ce10530" exitCode=0 Nov 03 22:43:11.518846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.518506 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" event={"ID":"ff4a701c-59e5-49be-a92b-76cab43d4d90","Type":"ContainerDied","Data":"a5e02f23aa879339b649c94d1c5c5710536d49a94379c1bf492284b36ce10530"} Nov 03 22:43:11.518846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.518588 4898 scope.go:122] "RemoveContainer" containerID="4d7b86a45d0023aa6a293952300bac25ecdc62921c316170cb9f6057ff82b589" Nov 03 22:43:11.583206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.582840 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:11.583206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.582903 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:11.809851 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.809827 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:43:11.926387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926353 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-token\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926449 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-podinfo\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926482 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-kube-api-access-mhhjr\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-kube-api-access-mhhjr\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926530 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-credential-socket\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926578 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-certs\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926607 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-envoy\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926634 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-data\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926728 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-socket\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.926887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.926775 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4a701c-59e5-49be-a92b-76cab43d4d90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a701c-59e5-49be-a92b-76cab43d4d90-istiod-ca-cert\") pod \"ff4a701c-59e5-49be-a92b-76cab43d4d90\" (UID: \"ff4a701c-59e5-49be-a92b-76cab43d4d90\") " Nov 03 22:43:11.927171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.927146 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-credential-socket" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.929114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.928503 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-socket" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.929114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.928612 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.929114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.928633 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.929578 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.929516 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-data" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.929578 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.929557 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-certs" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.929713 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.929615 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4a701c-59e5-49be-a92b-76cab43d4d90-istiod-ca-cert" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:11.929877 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.929853 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-token" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:11.930037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.929997 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-podinfo" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:11.967002 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.966933 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-kube-api-access-mhhjr" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "kube-api-access-mhhjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:11.973518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:11.973465 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-envoy" pod "ff4a701c-59e5-49be-a92b-76cab43d4d90" (UID: "ff4a701c-59e5-49be-a92b-76cab43d4d90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.030127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.030048 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.030127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.030086 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhhjr\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-kube-api-access-mhhjr\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.030127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.030099 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.030127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.030108 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.030127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.030117 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.030127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.030126 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a701c-59e5-49be-a92b-76cab43d4d90-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.030546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.030137 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a701c-59e5-49be-a92b-76cab43d4d90-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.182956 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.182921 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:43:12.231825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.231783 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-credential-socket\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.231860 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-certs\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.231889 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-data\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.231946 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11c9b845-2337-4d01-8b70-41443825c175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c9b845-2337-4d01-8b70-41443825c175-istio-podinfo\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.231991 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-istio-token\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-istio-token\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232054 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-credential-socket" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.232276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232127 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-socket\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232179 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-envoy\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232238 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-data" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.232463 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232323 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-kube-api-access-bpbtw\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-kube-api-access-bpbtw\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232463 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232378 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c9b845-2337-4d01-8b70-41443825c175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c9b845-2337-4d01-8b70-41443825c175-istiod-ca-cert\") pod \"11c9b845-2337-4d01-8b70-41443825c175\" (UID: \"11c9b845-2337-4d01-8b70-41443825c175\") " Nov 03 22:43:12.232463 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232421 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-certs" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.232590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232571 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-socket" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.232866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232849 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.232866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232869 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.233016 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232883 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.233016 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.232896 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.233506 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.233483 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c9b845-2337-4d01-8b70-41443825c175-istiod-ca-cert" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:12.276652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.276597 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-istio-token" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:12.277009 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.276974 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-kube-api-access-bpbtw" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "kube-api-access-bpbtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:12.280374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.280310 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-envoy" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.282213 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.282182 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11c9b845-2337-4d01-8b70-41443825c175-istio-podinfo" pod "11c9b845-2337-4d01-8b70-41443825c175" (UID: "11c9b845-2337-4d01-8b70-41443825c175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:12.360428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.360313 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c9b845-2337-4d01-8b70-41443825c175-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.360428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.360354 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbtw\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-kube-api-access-bpbtw\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.360428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.360372 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c9b845-2337-4d01-8b70-41443825c175-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.360428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.360386 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c9b845-2337-4d01-8b70-41443825c175-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.360428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.360401 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c9b845-2337-4d01-8b70-41443825c175-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.385212 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.385178 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461330 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-envoy\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461440 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-kube-api-access-dtfp6\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-kube-api-access-dtfp6\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461484 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-data\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461596 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0add3bd-313c-4c30-a50c-5639957343e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0add3bd-313c-4c30-a50c-5639957343e1-istiod-ca-cert\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461638 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0add3bd-313c-4c30-a50c-5639957343e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0add3bd-313c-4c30-a50c-5639957343e1-istio-podinfo\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461686 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-certs\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461772 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-credential-socket\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461824 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-istio-token\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-istio-token\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.462824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.461856 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-socket\") pod \"a0add3bd-313c-4c30-a50c-5639957343e1\" (UID: \"a0add3bd-313c-4c30-a50c-5639957343e1\") " Nov 03 22:43:12.475322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.470486 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-socket" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.475322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.475015 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-data" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.475322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.475190 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0add3bd-313c-4c30-a50c-5639957343e1-istiod-ca-cert" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:12.475322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.475250 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-certs" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.475322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.475297 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-credential-socket" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.479447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.479407 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-envoy" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.480969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.480929 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0add3bd-313c-4c30-a50c-5639957343e1-istio-podinfo" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:12.481386 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.481356 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-istio-token" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:12.481644 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.481616 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-kube-api-access-dtfp6" pod "a0add3bd-313c-4c30-a50c-5639957343e1" (UID: "a0add3bd-313c-4c30-a50c-5639957343e1"). InnerVolumeSpecName "kube-api-access-dtfp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:12.532428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.532343 4898 generic.go:356] "Generic (PLEG): container finished" podID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerID="313b5003e527a56e8d3d9a11a983d3ab8065351324f1e3bda474f803c32f12f6" exitCode=0 Nov 03 22:43:12.532880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.532536 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" event={"ID":"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3","Type":"ContainerDied","Data":"313b5003e527a56e8d3d9a11a983d3ab8065351324f1e3bda474f803c32f12f6"} Nov 03 22:43:12.578920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.573404 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.578920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.573461 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:12.578920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.577096 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" event={"ID":"ff4a701c-59e5-49be-a92b-76cab43d4d90","Type":"ContainerDied","Data":"90351c1aa952fc95dccdbc9c60a2d9788322d2ce0357170c3868299b2bead30f"} Nov 03 22:43:12.578920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.577159 4898 scope.go:122] "RemoveContainer" containerID="a5e02f23aa879339b649c94d1c5c5710536d49a94379c1bf492284b36ce10530" Nov 03 22:43:12.578920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.577277 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582072 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582097 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0add3bd-313c-4c30-a50c-5639957343e1-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582112 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0add3bd-313c-4c30-a50c-5639957343e1-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582129 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582144 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582160 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582175 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582189 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0add3bd-313c-4c30-a50c-5639957343e1-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.582292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.582203 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfp6\" (UniqueName: \"kubernetes.io/projected/a0add3bd-313c-4c30-a50c-5639957343e1-kube-api-access-dtfp6\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.586824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.584589 4898 generic.go:356] "Generic (PLEG): container finished" podID="a0add3bd-313c-4c30-a50c-5639957343e1" containerID="24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690" exitCode=0 Nov 03 22:43:12.586824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.584748 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" event={"ID":"a0add3bd-313c-4c30-a50c-5639957343e1","Type":"ContainerDied","Data":"24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690"} Nov 03 22:43:12.586824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.584812 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" event={"ID":"a0add3bd-313c-4c30-a50c-5639957343e1","Type":"ContainerDied","Data":"aac8975d9d6d229f43c758c95a444357f49cda044fd89d79a1255c8b0181d0c7"} Nov 03 22:43:12.586824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.584905 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x" Nov 03 22:43:12.595911 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.595587 4898 generic.go:356] "Generic (PLEG): container finished" podID="11c9b845-2337-4d01-8b70-41443825c175" containerID="da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c" exitCode=0 Nov 03 22:43:12.595911 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.595658 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" event={"ID":"11c9b845-2337-4d01-8b70-41443825c175","Type":"ContainerDied","Data":"da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c"} Nov 03 22:43:12.595911 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.595690 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" event={"ID":"11c9b845-2337-4d01-8b70-41443825c175","Type":"ContainerDied","Data":"14a44a7af6ddc0c1284e6457620afd400ba61c3d7831a63a0ce6249f4911407d"} Nov 03 22:43:12.595911 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.595745 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj" Nov 03 22:43:12.604685 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.604662 4898 scope.go:122] "RemoveContainer" containerID="24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690" Nov 03 22:43:12.616650 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.616633 4898 scope.go:122] "RemoveContainer" containerID="24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690" Nov 03 22:43:12.616935 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:12.616910 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690\": container with ID starting with 24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690 not found: ID does not exist" containerID="24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690" Nov 03 22:43:12.617044 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.616942 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690"} err="failed to get container status \"24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690\": rpc error: code = NotFound desc = could not find container \"24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690\": container with ID starting with 24d7311621cb8c28a167dd21a061b91b583155e808bc9a4fc1bcb5038e41e690 not found: ID does not exist" Nov 03 22:43:12.617044 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.616967 4898 scope.go:122] "RemoveContainer" containerID="da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c" Nov 03 22:43:12.627594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.627572 4898 scope.go:122] "RemoveContainer" containerID="da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c" Nov 03 22:43:12.627924 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:12.627891 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c\": container with ID starting with da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c not found: ID does not exist" containerID="da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c" Nov 03 22:43:12.627997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.627936 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c"} err="failed to get container status \"da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c\": rpc error: code = NotFound desc = could not find container \"da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c\": container with ID starting with da8293a96e00ca05c5124aefa827b418db7e1589c93774156c76a0162ec7175c not found: ID does not exist" Nov 03 22:43:12.718617 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.718583 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:43:12.771891 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.771824 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv"] Nov 03 22:43:12.775997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.775968 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-d5brv"] Nov 03 22:43:12.887043 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887007 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x"] Nov 03 22:43:12.887436 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887417 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-certs\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887557 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887451 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-token\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-token\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887557 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887514 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-envoy\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887557 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887537 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-podinfo\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887559 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-credential-socket\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887600 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istiod-ca-cert\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887620 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-socket\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887723 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887659 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-data\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.887959 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.887776 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-kube-api-access-v5j8h\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-kube-api-access-v5j8h\") pod \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\" (UID: \"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3\") " Nov 03 22:43:12.888502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.888351 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-certs" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.888734 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.888709 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istiod-ca-cert" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:12.889135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.889102 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-socket" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.889367 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.889340 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-credential-socket" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.890419 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.890383 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-data" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.891007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.890960 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-envoy" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:12.895841 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.895049 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-podinfo" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:12.895841 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.895115 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-kube-api-access-v5j8h" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "kube-api-access-v5j8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:12.896301 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.896276 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-token" pod "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" (UID: "2b7d2e54-78db-45b1-8a8b-97fbd35f19a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:12.896624 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.896604 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4vg6x"] Nov 03 22:43:12.907882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.907857 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" path="/var/lib/kubelet/pods/a0add3bd-313c-4c30-a50c-5639957343e1/volumes" Nov 03 22:43:12.908574 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.908554 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" path="/var/lib/kubelet/pods/ff4a701c-59e5-49be-a92b-76cab43d4d90/volumes" Nov 03 22:43:12.990167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990127 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990167 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990184 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990199 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990213 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990229 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990243 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990257 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5j8h\" (UniqueName: \"kubernetes.io/projected/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-kube-api-access-v5j8h\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.990427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:12.990272 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:13.021866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.021824 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj"] Nov 03 22:43:13.027517 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.027490 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ftxcj"] Nov 03 22:43:13.562812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.562265 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.562812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.562350 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:13.604152 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.604118 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" Nov 03 22:43:13.604325 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.604123 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct" event={"ID":"2b7d2e54-78db-45b1-8a8b-97fbd35f19a3","Type":"ContainerDied","Data":"eff1ac122a0caeb5a524ffc0cffa03fe341c33e118775ee3ef24f73651060d62"} Nov 03 22:43:13.604397 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.604348 4898 scope.go:122] "RemoveContainer" containerID="313b5003e527a56e8d3d9a11a983d3ab8065351324f1e3bda474f803c32f12f6" Nov 03 22:43:13.776273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.776183 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct"] Nov 03 22:43:13.780727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:13.780702 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j72ct"] Nov 03 22:43:14.564864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:14.561230 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:14.564864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:14.561307 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:14.915234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:14.915199 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c9b845-2337-4d01-8b70-41443825c175" path="/var/lib/kubelet/pods/11c9b845-2337-4d01-8b70-41443825c175/volumes" Nov 03 22:43:14.916250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:14.916222 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" path="/var/lib/kubelet/pods/2b7d2e54-78db-45b1-8a8b-97fbd35f19a3/volumes" Nov 03 22:43:15.560724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:15.560688 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:43:15.560946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:15.560749 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:43:16.572811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:16.572178 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:16.572811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:16.572745 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:16.574034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:16.574011 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:43:19.836884 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:19.836842 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:43:22.245968 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.245935 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-4c198d"] Nov 03 22:43:22.246518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246505 4898 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" Nov 03 22:43:22.246518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246518 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" Nov 03 22:43:22.246600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246540 4898 cpu_manager.go:414] "containerMap: removing container" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" Nov 03 22:43:22.246600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246545 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" Nov 03 22:43:22.246600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246553 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" Nov 03 22:43:22.246600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246558 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" Nov 03 22:43:22.246600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246570 4898 cpu_manager.go:414] "containerMap: removing container" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" Nov 03 22:43:22.246600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246575 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" Nov 03 22:43:22.246771 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246666 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c9b845-2337-4d01-8b70-41443825c175" containerName="istio-proxy" Nov 03 22:43:22.246771 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246677 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0add3bd-313c-4c30-a50c-5639957343e1" containerName="istio-proxy" Nov 03 22:43:22.246771 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246684 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7d2e54-78db-45b1-8a8b-97fbd35f19a3" containerName="istio-proxy" Nov 03 22:43:22.246771 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.246691 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" Nov 03 22:43:22.251665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.251650 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:22.304581 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.304354 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-4c198d"] Nov 03 22:43:22.308575 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.308545 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxkv\" (UniqueName: \"kubernetes.io/projected/850a4b9c-88ff-4277-9143-e1d498f339fa-kube-api-access-ddxkv\") pod \"test-curl-7240-4c198d\" (UID: \"850a4b9c-88ff-4277-9143-e1d498f339fa\") " pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:22.349972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.349943 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j4jqv\"" Nov 03 22:43:22.409522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.409482 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxkv\" (UniqueName: \"kubernetes.io/projected/850a4b9c-88ff-4277-9143-e1d498f339fa-kube-api-access-ddxkv\") pod \"test-curl-7240-4c198d\" (UID: \"850a4b9c-88ff-4277-9143-e1d498f339fa\") " pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:22.471055 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.471025 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxkv\" (UniqueName: \"kubernetes.io/projected/850a4b9c-88ff-4277-9143-e1d498f339fa-kube-api-access-ddxkv\") pod \"test-curl-7240-4c198d\" (UID: \"850a4b9c-88ff-4277-9143-e1d498f339fa\") " pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:22.563673 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.563599 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:22.949136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:22.949112 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-4c198d"] Nov 03 22:43:22.951126 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:22.951089 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod850a4b9c_88ff_4277_9143_e1d498f339fa.slice/crio-51c0de8bcd9dae11013fdd48380c0a93ce59fa336a6aaf8ed779b50d1c03ded4 WatchSource:0}: Error finding container 51c0de8bcd9dae11013fdd48380c0a93ce59fa336a6aaf8ed779b50d1c03ded4: Status 404 returned error can't find the container with id 51c0de8bcd9dae11013fdd48380c0a93ce59fa336a6aaf8ed779b50d1c03ded4 Nov 03 22:43:23.673922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:23.673883 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-4c198d" event={"ID":"850a4b9c-88ff-4277-9143-e1d498f339fa","Type":"ContainerStarted","Data":"51c0de8bcd9dae11013fdd48380c0a93ce59fa336a6aaf8ed779b50d1c03ded4"} Nov 03 22:43:24.683479 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:24.683447 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-4c198d" event={"ID":"850a4b9c-88ff-4277-9143-e1d498f339fa","Type":"ContainerStarted","Data":"6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c"} Nov 03 22:43:24.797626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:24.797565 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-4c198d" podStartSLOduration=1.656275602 podStartE2EDuration="2.797551798s" totalImagesPullingTime="1.141276196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:24.796974041 +0000 UTC m=+1248.009743112" watchObservedRunningTime="2026-11-03 22:43:24.797551798 +0000 UTC m=+1248.010320849" Nov 03 22:43:26.261408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.261372 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-4c198d"] Nov 03 22:43:26.265424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.265394 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-4c198d"] Nov 03 22:43:26.424902 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.424865 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-79532f"] Nov 03 22:43:26.425484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.425469 4898 cpu_manager.go:414] "containerMap: removing container" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" containerName="test-curl-7240-4c198d" Nov 03 22:43:26.425546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.425484 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" containerName="test-curl-7240-4c198d" Nov 03 22:43:26.425546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.425498 4898 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" Nov 03 22:43:26.425546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.425503 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" Nov 03 22:43:26.425698 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.425603 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" containerName="test-curl-7240-4c198d" Nov 03 22:43:26.425698 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.425618 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a701c-59e5-49be-a92b-76cab43d4d90" containerName="istio-proxy" Nov 03 22:43:26.429066 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.429046 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:26.461003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.460974 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz64t\" (UniqueName: \"kubernetes.io/projected/5340d852-4941-47b7-b960-63a635a34fc0-kube-api-access-lz64t\") pod \"test-curl-7240-79532f\" (UID: \"5340d852-4941-47b7-b960-63a635a34fc0\") " pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:26.483069 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.483034 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-79532f"] Nov 03 22:43:26.562127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.562040 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz64t\" (UniqueName: \"kubernetes.io/projected/5340d852-4941-47b7-b960-63a635a34fc0-kube-api-access-lz64t\") pod \"test-curl-7240-79532f\" (UID: \"5340d852-4941-47b7-b960-63a635a34fc0\") " pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:26.623128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.623098 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz64t\" (UniqueName: \"kubernetes.io/projected/5340d852-4941-47b7-b960-63a635a34fc0-kube-api-access-lz64t\") pod \"test-curl-7240-79532f\" (UID: \"5340d852-4941-47b7-b960-63a635a34fc0\") " pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:26.696497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.696461 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-4c198d" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" containerName="test-curl-7240-4c198d" containerID="cri-o://6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c" gracePeriod=2 Nov 03 22:43:26.743976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:26.743939 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:26.748190 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:26.748157 4898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-4c198d\" is forbidden: User \"system:node:ip-10-0-139-173.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-173.us-west-2.compute.internal' and this object" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:26.957287 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:26.957242 4898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-4c198d\" is forbidden: User \"system:node:ip-10-0-139-173.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-173.us-west-2.compute.internal' and this object" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:27.129965 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:27.129933 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-79532f"] Nov 03 22:43:27.131318 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:27.131286 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5340d852_4941_47b7_b960_63a635a34fc0.slice/crio-18add46719101152b1a33468f8f6aa1c333f4c99f1c18960057f20c962af7442 WatchSource:0}: Error finding container 18add46719101152b1a33468f8f6aa1c333f4c99f1c18960057f20c962af7442: Status 404 returned error can't find the container with id 18add46719101152b1a33468f8f6aa1c333f4c99f1c18960057f20c962af7442 Nov 03 22:43:27.704619 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:27.704585 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-79532f" event={"ID":"5340d852-4941-47b7-b960-63a635a34fc0","Type":"ContainerStarted","Data":"66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8"} Nov 03 22:43:27.705045 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:27.704632 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-79532f" event={"ID":"5340d852-4941-47b7-b960-63a635a34fc0","Type":"ContainerStarted","Data":"18add46719101152b1a33468f8f6aa1c333f4c99f1c18960057f20c962af7442"} Nov 03 22:43:27.820062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:27.820007 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-79532f" podStartSLOduration=1.819992991 podStartE2EDuration="1.819992991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:27.817848501 +0000 UTC m=+1251.030617575" watchObservedRunningTime="2026-11-03 22:43:27.819992991 +0000 UTC m=+1251.032762085" Nov 03 22:43:28.324132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.324101 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p"] Nov 03 22:43:28.328336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.328315 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.374544 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.374515 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5"] Nov 03 22:43:28.378347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.378328 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.382425 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382401 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.382522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382443 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.382522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382460 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.382646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382621 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lpj\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-kube-api-access-m2lpj\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.382775 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382757 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.382885 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382869 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.382929 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382903 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.382969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.382959 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.383022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.383008 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.467639 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.467602 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5"] Nov 03 22:43:28.484277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484230 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.484277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484278 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.484520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484368 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.484520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484407 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.484520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484431 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.484520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484470 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.484520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484505 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.484836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484772 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.484913 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484846 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.484913 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.484898 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.485106 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485075 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hbf\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-kube-api-access-47hbf\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.485231 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485140 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.485231 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485172 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.485385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485286 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.485874 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485848 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.485964 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485886 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.485964 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485942 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.486082 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.485984 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lpj\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-kube-api-access-m2lpj\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.486166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.486133 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.486225 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.486195 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.486281 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.486227 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.486839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.486660 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.486839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.486781 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.488856 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.488769 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.488944 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.488924 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.536776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.536742 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p"] Nov 03 22:43:28.548931 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.548864 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lpj\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-kube-api-access-m2lpj\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.550211 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.550182 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.587189 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587094 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587189 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587151 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hbf\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-kube-api-access-47hbf\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587189 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587185 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587233 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587307 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587340 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587366 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587394 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.587480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587418 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.588258 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.587968 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.588258 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.588053 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.588258 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.588222 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.588860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.588835 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.589469 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.589426 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.591172 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.591152 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.591284 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.591241 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.642283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.642242 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:28.651083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.651057 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.651269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.651244 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hbf\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-kube-api-access-47hbf\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.695756 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.694264 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:28.955742 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:28.955715 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:29.078131 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.078102 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p"] Nov 03 22:43:29.093940 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.093909 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850a4b9c-88ff-4277-9143-e1d498f339fa-kube-api-access-ddxkv\" (UniqueName: \"kubernetes.io/projected/850a4b9c-88ff-4277-9143-e1d498f339fa-kube-api-access-ddxkv\") pod \"850a4b9c-88ff-4277-9143-e1d498f339fa\" (UID: \"850a4b9c-88ff-4277-9143-e1d498f339fa\") " Nov 03 22:43:29.096269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.096235 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850a4b9c-88ff-4277-9143-e1d498f339fa-kube-api-access-ddxkv" pod "850a4b9c-88ff-4277-9143-e1d498f339fa" (UID: "850a4b9c-88ff-4277-9143-e1d498f339fa"). InnerVolumeSpecName "kube-api-access-ddxkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:29.113805 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:29.113763 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda638b3fc_f541_4ab3_b0ad_7fd36574b367.slice/crio-fd4141a636af3c823cc5f859122a3237610511a6a6af597ea44b48780accdb7a WatchSource:0}: Error finding container fd4141a636af3c823cc5f859122a3237610511a6a6af597ea44b48780accdb7a: Status 404 returned error can't find the container with id fd4141a636af3c823cc5f859122a3237610511a6a6af597ea44b48780accdb7a Nov 03 22:43:29.116387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.116349 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.116489 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.116436 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.116489 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.116480 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.130651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.130626 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5"] Nov 03 22:43:29.130884 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:29.130853 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f07358_7c7d_45f1_ac0a_7884d16bfca4.slice/crio-4ed7b9b07aaa252c25c0e0a172cfda75b775fb01664c8267fd74c612fa8b86eb WatchSource:0}: Error finding container 4ed7b9b07aaa252c25c0e0a172cfda75b775fb01664c8267fd74c612fa8b86eb: Status 404 returned error can't find the container with id 4ed7b9b07aaa252c25c0e0a172cfda75b775fb01664c8267fd74c612fa8b86eb Nov 03 22:43:29.133419 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.133387 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.133503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.133455 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.133503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.133485 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.195483 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.195455 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxkv\" (UniqueName: \"kubernetes.io/projected/850a4b9c-88ff-4277-9143-e1d498f339fa-kube-api-access-ddxkv\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:29.307371 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.307335 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-79532f"] Nov 03 22:43:29.311514 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.311486 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-79532f"] Nov 03 22:43:29.723887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.723847 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" event={"ID":"a638b3fc-f541-4ab3-b0ad-7fd36574b367","Type":"ContainerStarted","Data":"0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57"} Nov 03 22:43:29.724091 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.723904 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" event={"ID":"a638b3fc-f541-4ab3-b0ad-7fd36574b367","Type":"ContainerStarted","Data":"fd4141a636af3c823cc5f859122a3237610511a6a6af597ea44b48780accdb7a"} Nov 03 22:43:29.725497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.725470 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" event={"ID":"12f07358-7c7d-45f1-ac0a-7884d16bfca4","Type":"ContainerStarted","Data":"3adb356a35cbedbc720ebc4757f8e59399ad614bfc1e5f2b6ffe9a5a3a5bf1a4"} Nov 03 22:43:29.725688 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.725665 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" event={"ID":"12f07358-7c7d-45f1-ac0a-7884d16bfca4","Type":"ContainerStarted","Data":"4ed7b9b07aaa252c25c0e0a172cfda75b775fb01664c8267fd74c612fa8b86eb"} Nov 03 22:43:29.726895 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.726873 4898 generic.go:356] "Generic (PLEG): container finished" podID="850a4b9c-88ff-4277-9143-e1d498f339fa" containerID="6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c" exitCode=137 Nov 03 22:43:29.726992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.726906 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:29.726992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.726947 4898 scope.go:122] "RemoveContainer" containerID="6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c" Nov 03 22:43:29.727227 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.727202 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-79532f" podUID="5340d852-4941-47b7-b960-63a635a34fc0" containerName="test-curl-7240-79532f" containerID="cri-o://66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8" gracePeriod=2 Nov 03 22:43:29.792893 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.792710 4898 scope.go:122] "RemoveContainer" containerID="6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c" Nov 03 22:43:29.793328 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:29.793290 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c\": container with ID starting with 6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c not found: ID does not exist" containerID="6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c" Nov 03 22:43:29.793510 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.793341 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c"} err="failed to get container status \"6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c\": rpc error: code = NotFound desc = could not find container \"6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c\": container with ID starting with 6060e8228062107c0420d0998e6850c6e9b70d9b7ac16b18b0c4213492288d5c not found: ID does not exist" Nov 03 22:43:29.997296 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.997226 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podStartSLOduration=1.997205631 podStartE2EDuration="1.997205631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:29.866168329 +0000 UTC m=+1253.078937415" watchObservedRunningTime="2026-11-03 22:43:29.997205631 +0000 UTC m=+1253.209974686" Nov 03 22:43:29.999868 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:29.998780 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podStartSLOduration=1.9987690759999999 podStartE2EDuration="1.998769076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:29.996176628 +0000 UTC m=+1253.208945700" watchObservedRunningTime="2026-11-03 22:43:29.998769076 +0000 UTC m=+1253.211538129" Nov 03 22:43:30.056171 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:30.056063 4898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-4c198d\" is forbidden: User \"system:node:ip-10-0-139-173.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-173.us-west-2.compute.internal' and this object" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" pod="opendatahub/test-curl-7240-4c198d" Nov 03 22:43:30.109209 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:30.107872 4898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-79532f\" is forbidden: User \"system:node:ip-10-0-139-173.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-173.us-west-2.compute.internal' and this object" podUID="5340d852-4941-47b7-b960-63a635a34fc0" pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:30.660811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:30.654868 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:30.680812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:30.676914 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:43:30.680812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:30.677003 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 22:43:30.699842 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:30.698171 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:30.700681 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:30.700616 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:43:30.700841 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:30.700700 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 22:43:30.912556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:30.912471 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850a4b9c-88ff-4277-9143-e1d498f339fa" path="/var/lib/kubelet/pods/850a4b9c-88ff-4277-9143-e1d498f339fa/volumes" Nov 03 22:43:31.656811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:31.650214 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:43:31.656811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:31.650279 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 22:43:31.696023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:31.695989 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:43:31.696194 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:31.696039 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 22:43:32.127613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.127587 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:32.299874 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.299494 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5340d852-4941-47b7-b960-63a635a34fc0-kube-api-access-lz64t\" (UniqueName: \"kubernetes.io/projected/5340d852-4941-47b7-b960-63a635a34fc0-kube-api-access-lz64t\") pod \"5340d852-4941-47b7-b960-63a635a34fc0\" (UID: \"5340d852-4941-47b7-b960-63a635a34fc0\") " Nov 03 22:43:32.301754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.301731 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5340d852-4941-47b7-b960-63a635a34fc0-kube-api-access-lz64t" pod "5340d852-4941-47b7-b960-63a635a34fc0" (UID: "5340d852-4941-47b7-b960-63a635a34fc0"). InnerVolumeSpecName "kube-api-access-lz64t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:32.401454 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.401416 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz64t\" (UniqueName: \"kubernetes.io/projected/5340d852-4941-47b7-b960-63a635a34fc0-kube-api-access-lz64t\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:32.653946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.653177 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:43:32.653946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.653237 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 22:43:32.695393 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.695360 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:43:32.695779 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.695413 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 22:43:32.794412 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.794370 4898 generic.go:356] "Generic (PLEG): container finished" podID="5340d852-4941-47b7-b960-63a635a34fc0" containerID="66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8" exitCode=137 Nov 03 22:43:32.794911 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.794888 4898 scope.go:122] "RemoveContainer" containerID="66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8" Nov 03 22:43:32.795176 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.794974 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:32.807114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.807091 4898 scope.go:122] "RemoveContainer" containerID="66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8" Nov 03 22:43:32.808322 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:32.808294 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8\": container with ID starting with 66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8 not found: ID does not exist" containerID="66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8" Nov 03 22:43:32.808429 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.808329 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8"} err="failed to get container status \"66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8\": rpc error: code = NotFound desc = could not find container \"66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8\": container with ID starting with 66680f023b5f19d85fc0bda6a7283b32c17c7d3022852e59ce931236e447d8e8 not found: ID does not exist" Nov 03 22:43:32.861120 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:32.861050 4898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-79532f\" is forbidden: User \"system:node:ip-10-0-139-173.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-173.us-west-2.compute.internal' and this object" podUID="5340d852-4941-47b7-b960-63a635a34fc0" pod="opendatahub/test-curl-7240-79532f" Nov 03 22:43:32.916260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:32.916168 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5340d852-4941-47b7-b960-63a635a34fc0" path="/var/lib/kubelet/pods/5340d852-4941-47b7-b960-63a635a34fc0/volumes" Nov 03 22:43:33.662815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:33.661820 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:43:33.662815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:33.661884 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 22:43:33.695433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:33.695394 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:43:33.695894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:33.695450 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 22:43:34.471809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.471419 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5"] Nov 03 22:43:34.489229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.472364 4898 cpu_manager.go:414] "containerMap: removing container" podUID="5340d852-4941-47b7-b960-63a635a34fc0" containerName="test-curl-7240-79532f" Nov 03 22:43:34.489229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.472388 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5340d852-4941-47b7-b960-63a635a34fc0" containerName="test-curl-7240-79532f" Nov 03 22:43:34.489229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.472553 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5340d852-4941-47b7-b960-63a635a34fc0" containerName="test-curl-7240-79532f" Nov 03 22:43:34.493739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.492948 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.533349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533320 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.533490 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533440 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.533545 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533511 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.533601 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533584 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.533694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533675 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.533749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533735 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.537820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533868 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.537820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533920 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.537820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.533947 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj5t2\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-kube-api-access-cj5t2\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.615240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.614265 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5"] Nov 03 22:43:34.676879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.642723 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.684812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.677156 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.696812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.691958 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.697392 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.697357 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.705901 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.652488 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:43:34.706134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.706103 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 22:43:34.706265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.705944 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.706425 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.706406 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.706555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.706540 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.706661 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.706646 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj5t2\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-kube-api-access-cj5t2\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.706872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.706851 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.707199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.707170 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.707291 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.652181 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.707291 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.707258 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:43:34.707410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.707296 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 22:43:34.708038 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.707987 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.708152 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.708065 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.708271 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.708249 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.710573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.710553 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.711353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.711328 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.800284 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.800202 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj5t2\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-kube-api-access-cj5t2\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.803204 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.803176 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.803337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.803297 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz"] Nov 03 22:43:34.810586 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.809140 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.814690 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.814665 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k"] Nov 03 22:43:34.816242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.816221 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:34.820652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.820630 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.884134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.878270 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz"] Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918602 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqhw\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-kube-api-access-bcqhw\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918635 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918665 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918690 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918725 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918779 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918821 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918854 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918898 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918927 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918946 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.918983 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.919008 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.919026 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.919055 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.919087 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.919104 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.923098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.919135 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhmd\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-kube-api-access-4lhmd\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:34.929408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:34.929384 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k"] Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020261 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020322 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020379 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020420 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020444 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020487 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020521 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020546 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020579 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020621 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020648 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020675 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhmd\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-kube-api-access-4lhmd\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020734 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqhw\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-kube-api-access-bcqhw\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020760 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020825 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020863 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.020923 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.021448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.021013 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.022913 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.022892 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.023166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.023140 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.023953 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.023505 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.024757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.024730 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.025451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.025425 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.025691 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.025669 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.102470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.032380 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.102470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.033067 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.102470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.033448 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.102470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.033605 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.102470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.033726 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.115137 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.111919 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.115838 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.115750 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.116735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.116691 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.117106 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.116936 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqhw\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-kube-api-access-bcqhw\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.117106 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.117071 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.118942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.118917 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhmd\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-kube-api-access-4lhmd\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.122262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.122195 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.312606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.312564 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:35.315682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.315659 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:35.405451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.404311 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5"] Nov 03 22:43:35.427187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.426843 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.427816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.427339 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.427816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.427398 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.644842 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.643259 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:43:35.644842 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.643316 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 22:43:35.695265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.695187 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:43:35.695265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.695241 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 22:43:35.726591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.726503 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k"] Nov 03 22:43:35.727079 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:35.726730 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985df9f5_baaa_48c6_b7a2_438ddcb8b3e1.slice/crio-cc8bc4338c7bf5f8c0ca414fe9d9840c2ff97888c2c4957c7a38aa5c697d53fa WatchSource:0}: Error finding container cc8bc4338c7bf5f8c0ca414fe9d9840c2ff97888c2c4957c7a38aa5c697d53fa: Status 404 returned error can't find the container with id cc8bc4338c7bf5f8c0ca414fe9d9840c2ff97888c2c4957c7a38aa5c697d53fa Nov 03 22:43:35.730105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.730061 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.730234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.730172 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.730234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.730216 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.787046 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.785203 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz"] Nov 03 22:43:35.787046 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:35.785811 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b69acfc_5ad4_4a29_8d1a_f3e555f04cb0.slice/crio-53b2523ab7870883ae5a4f4417197907d4e47c8f34969376762e9827a8d16cb1 WatchSource:0}: Error finding container 53b2523ab7870883ae5a4f4417197907d4e47c8f34969376762e9827a8d16cb1: Status 404 returned error can't find the container with id 53b2523ab7870883ae5a4f4417197907d4e47c8f34969376762e9827a8d16cb1 Nov 03 22:43:35.791749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.791707 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.792007 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.791979 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.792170 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.792147 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.826728 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.826689 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" event={"ID":"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0","Type":"ContainerStarted","Data":"53b2523ab7870883ae5a4f4417197907d4e47c8f34969376762e9827a8d16cb1"} Nov 03 22:43:35.829941 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.829879 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" event={"ID":"d67caae8-c0f7-432a-bc11-7e72cb5750c4","Type":"ContainerStarted","Data":"9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052"} Nov 03 22:43:35.830114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.829952 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" event={"ID":"d67caae8-c0f7-432a-bc11-7e72cb5750c4","Type":"ContainerStarted","Data":"867997befd2bf81fea5e9dbb410d61dfbe4f3c7158d808cee6a23d42b0a9b6e4"} Nov 03 22:43:35.831633 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:35.831574 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" event={"ID":"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1","Type":"ContainerStarted","Data":"cc8bc4338c7bf5f8c0ca414fe9d9840c2ff97888c2c4957c7a38aa5c697d53fa"} Nov 03 22:43:36.822812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:36.822082 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:36.822812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:36.822256 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:36.822812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:36.822303 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:36.893317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:36.893275 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" event={"ID":"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0","Type":"ContainerStarted","Data":"a41330f97148284f490910856e8ceb91b0f73740c8d2378e60b3fca55e232c07"} Nov 03 22:43:36.897958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:36.897875 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" event={"ID":"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1","Type":"ContainerStarted","Data":"9ee0438ac8f030465e7dee929457e9742a1082af3e88a18f752eefe258668151"} Nov 03 22:43:37.321844 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.308717 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podStartSLOduration=3.308693124 podStartE2EDuration="3.308693124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:35.975663893 +0000 UTC m=+1259.188432956" watchObservedRunningTime="2026-11-03 22:43:37.308693124 +0000 UTC m=+1260.521462176" Nov 03 22:43:37.525840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.512280 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:37.525840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.512355 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:37.525840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.512636 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:37.525840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.512672 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:37.630872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.565870 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:37.630872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.565945 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:37.630872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.566017 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:37.630872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.566075 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:37.638975 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.638943 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:37.639206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.639193 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:37.645010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.644272 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:43:37.735813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.727160 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:43:37.816930 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.804308 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podStartSLOduration=3.80428819 podStartE2EDuration="3.80428819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:37.799144945 +0000 UTC m=+1261.011914008" watchObservedRunningTime="2026-11-03 22:43:37.80428819 +0000 UTC m=+1261.017057241" Nov 03 22:43:37.830227 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.830154 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podStartSLOduration=3.830131266 podStartE2EDuration="3.830131266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:37.348024039 +0000 UTC m=+1260.560793116" watchObservedRunningTime="2026-11-03 22:43:37.830131266 +0000 UTC m=+1261.042900319" Nov 03 22:43:37.861231 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.861202 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:37.880456 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:37.880417 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:38.399989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:38.390446 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:38.399989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:38.390511 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:38.399989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:38.391083 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:38.399989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:38.391124 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:38.877392 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:38.831102 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:38.877392 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:38.831175 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:39.319825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:39.313401 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:39.319825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:39.313463 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:39.319825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:39.317118 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:39.319825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:39.317170 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:39.962972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:39.962489 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:39.962972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:39.962548 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:40.324874 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.314267 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:40.324874 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.314332 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:40.324874 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.321117 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:40.324874 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.321172 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:40.467813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.466848 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4"] Nov 03 22:43:40.552567 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.552532 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4"] Nov 03 22:43:40.552904 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.552887 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.614393 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.614276 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-027a6fbe-gw-openshift-default-dockercfg-zfkbh\"" Nov 03 22:43:40.791840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.781140 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-credential-socket\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.792088 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.791847 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-socket\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.792088 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.792021 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-podinfo\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.792218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.792178 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-certs\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.819812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.792403 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-envoy\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.819812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.804860 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-data\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.819812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.804916 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istiod-ca-cert\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.819812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.805045 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-token\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.819812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.805081 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nngx\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-kube-api-access-7nngx\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.859414 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.858895 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:40.859414 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.858961 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.908598 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istiod-ca-cert\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.908705 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-token\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.908740 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nngx\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-kube-api-access-7nngx\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.908783 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-credential-socket\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.908838 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-socket\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.908906 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-podinfo\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.908985 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-certs\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.909107 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-envoy\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.909132 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-data\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.909564 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-data\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.910224 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istiod-ca-cert\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.910953 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-credential-socket\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.916972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.911195 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-socket\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.991814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.963047 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-certs\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.992370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.986919 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-podinfo\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:40.992488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:40.987810 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-envoy\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:41.003603 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.003537 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nngx\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-kube-api-access-7nngx\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:41.046718 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.046685 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-token\") pod \"e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:41.294154 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.294057 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:41.360546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.360509 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:41.360816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.360771 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:41.475811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.475136 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:41.475811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.475198 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:41.936811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.934057 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:41.936811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:41.934125 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:42.396571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.396024 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:42.396571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.396087 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:42.396571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.396490 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:42.396571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.396521 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:42.742818 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.735258 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4"] Nov 03 22:43:42.840283 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:42.836212 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c6a4e6_2b93_4f19_9e94_cf4fddd4d5d8.slice/crio-40f5226a9dc5d99739d93f016192ba3ce93b44c01a3d266a217c58a152cba96a WatchSource:0}: Error finding container 40f5226a9dc5d99739d93f016192ba3ce93b44c01a3d266a217c58a152cba96a: Status 404 returned error can't find the container with id 40f5226a9dc5d99739d93f016192ba3ce93b44c01a3d266a217c58a152cba96a Nov 03 22:43:42.840283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.836294 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:42.840283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.836337 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:43.020863 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.969514 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:43.020863 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.969677 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:43.020863 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:42.969768 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:43.193812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.190550 4898 generic.go:356] "Generic (PLEG): container finished" podID="78156a15-e7f0-4de1-a254-20e2e4a16100" containerID="905ac8fc9e4bf28cfaa0b06faae8931fdc29f71797682c08e257daa28081b171" exitCode=0 Nov 03 22:43:43.193812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.190914 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" event={"ID":"78156a15-e7f0-4de1-a254-20e2e4a16100","Type":"ContainerDied","Data":"905ac8fc9e4bf28cfaa0b06faae8931fdc29f71797682c08e257daa28081b171"} Nov 03 22:43:43.232339 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.232291 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" event={"ID":"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8","Type":"ContainerStarted","Data":"40f5226a9dc5d99739d93f016192ba3ce93b44c01a3d266a217c58a152cba96a"} Nov 03 22:43:43.319356 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.313427 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:43.319356 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.313487 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:43.327087 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.326962 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:43.327087 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.327023 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:43.816682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.816563 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:43.816682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:43.816618 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:44.272681 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.272600 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" event={"ID":"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8","Type":"ContainerStarted","Data":"faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2"} Nov 03 22:43:44.300107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.298301 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:44.300684 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.300380 4898 patch_prober.go:29] interesting pod/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:43:44.300684 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.300432 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 22:43:44.320707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.319846 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:44.320707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.319904 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:44.321135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.321041 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:44.321135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.321093 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:44.421862 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.421799 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podStartSLOduration=4.421764766 podStartE2EDuration="4.421764766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:44.405180386 +0000 UTC m=+1267.617949465" watchObservedRunningTime="2026-11-03 22:43:44.421764766 +0000 UTC m=+1267.634533818" Nov 03 22:43:44.884847 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.855732 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:43:44.886626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.886583 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:43:44.987542 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:44.987504 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:43:45.065431 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.065391 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-service-ca-bundle\") pod \"78156a15-e7f0-4de1-a254-20e2e4a16100\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " Nov 03 22:43:45.065686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.065523 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78156a15-e7f0-4de1-a254-20e2e4a16100-kube-api-access-n9bpd\" (UniqueName: \"kubernetes.io/projected/78156a15-e7f0-4de1-a254-20e2e4a16100-kube-api-access-n9bpd\") pod \"78156a15-e7f0-4de1-a254-20e2e4a16100\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " Nov 03 22:43:45.065686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.065555 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-trusted-ca-bundle\") pod \"78156a15-e7f0-4de1-a254-20e2e4a16100\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " Nov 03 22:43:45.065867 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.065717 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78156a15-e7f0-4de1-a254-20e2e4a16100-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/78156a15-e7f0-4de1-a254-20e2e4a16100-archives-path\") pod \"78156a15-e7f0-4de1-a254-20e2e4a16100\" (UID: \"78156a15-e7f0-4de1-a254-20e2e4a16100\") " Nov 03 22:43:45.065934 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.065899 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-service-ca-bundle" pod "78156a15-e7f0-4de1-a254-20e2e4a16100" (UID: "78156a15-e7f0-4de1-a254-20e2e4a16100"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:45.067246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.066420 4898 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-service-ca-bundle\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:45.067246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.067202 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78156a15-e7f0-4de1-a254-20e2e4a16100-archives-path" pod "78156a15-e7f0-4de1-a254-20e2e4a16100" (UID: "78156a15-e7f0-4de1-a254-20e2e4a16100"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:45.067840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.067816 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-trusted-ca-bundle" pod "78156a15-e7f0-4de1-a254-20e2e4a16100" (UID: "78156a15-e7f0-4de1-a254-20e2e4a16100"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:45.070283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.070256 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78156a15-e7f0-4de1-a254-20e2e4a16100-kube-api-access-n9bpd" pod "78156a15-e7f0-4de1-a254-20e2e4a16100" (UID: "78156a15-e7f0-4de1-a254-20e2e4a16100"). InnerVolumeSpecName "kube-api-access-n9bpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:45.168866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.167996 4898 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/78156a15-e7f0-4de1-a254-20e2e4a16100-archives-path\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:45.168866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.168032 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9bpd\" (UniqueName: \"kubernetes.io/projected/78156a15-e7f0-4de1-a254-20e2e4a16100-kube-api-access-n9bpd\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:45.168866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.168049 4898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78156a15-e7f0-4de1-a254-20e2e4a16100-trusted-ca-bundle\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:45.299867 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.299155 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" Nov 03 22:43:45.300046 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.299907 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dwnbd-4pz2h" event={"ID":"78156a15-e7f0-4de1-a254-20e2e4a16100","Type":"ContainerDied","Data":"dfa6f8398ef290ae31f680182c67a6af90c92901ddbcc98c423fc85594bc751b"} Nov 03 22:43:45.300149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.300054 4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa6f8398ef290ae31f680182c67a6af90c92901ddbcc98c423fc85594bc751b" Nov 03 22:43:45.301825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.301296 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:45.313942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.313916 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:45.314061 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.313967 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:45.316303 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.316280 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:43:45.316424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:45.316326 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:43:46.329180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:46.326275 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:46.329180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:46.327281 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:43:46.329180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:46.327328 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 22:43:46.340521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:46.340480 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:43:46.935811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:46.921068 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:46.936140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:46.936099 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.345924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:47.345880 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:47.346538 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:47.346502 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.491127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:47.485920 4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.969851 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:47.966474 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:47.969851 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:47.966564 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.012019 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.011950 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.176686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.176632 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.241522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.241421 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:43:48.270156 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.249908 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.374057 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.373301 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:48.374057 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.373376 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.392419 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.392365 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:48.392623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:48.392438 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.032142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.032094 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.048865 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.033327 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:43:49.049720 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.049687 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:49.049983 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.049953 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.183110 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.182970 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:49.188826 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.187213 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:43:49.335938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.289334 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.335938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.289766 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:43:49.373881 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.351507 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:43:49.386248 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.386213 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:43:49.409845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.408883 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:49.409845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.408957 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.670890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.652935 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:49.670890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.652999 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.784068 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.783930 4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.838133 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:49.838093 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:50.074877 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.068977 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.110823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.109941 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:50.110823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.110008 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.388945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.374921 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.388945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.375704 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:50.388945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.382353 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:43:50.388945 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.382695 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:50.393253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.393219 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:43:50.544814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.541884 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:50.544814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.541954 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.696062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.695961 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:50.696062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:50.696034 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.799515 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:43:50.799474 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:43:51.108953 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.107930 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 22:43:51.153058 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.152899 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:51.153058 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.152969 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:51.153767 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.153378 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:51.153767 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.153425 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:51.435866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.419892 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:51.435866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.420317 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:51.435866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.420585 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:51.613940 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.613898 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:51.614193 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.614161 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:51.748107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.748012 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:51.748352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:51.748321 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.088868 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.066572 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:43:52.152837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.148882 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:43:52.210015 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.209946 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:52.211249 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.211173 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.468074 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.467961 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:43:52.814556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.491547 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:52.952076 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.743334 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:52.952076 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.948448 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.958584 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.808629 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:52.958865 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:52.958830 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.083833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:53.079890 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.388126 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:53.388017 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.430854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:53.427884 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:53.430854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:53.427966 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.571823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:53.563764 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.986813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:53.961295 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:54.039076 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.015215 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:54.040391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.040031 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:54.040391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.040099 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:54.040720 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.040675 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:54.065823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.053922 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:54.065823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.057983 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:43:54.092809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.088947 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 22:43:54.092809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.089672 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:43:54.398818 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.392048 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:43:54.407167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.405955 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:43:54.602809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.598851 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:54.884490 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.884441 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:54.891937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.891911 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:43:54.920187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.919991 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:43:54.972305 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.972266 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:54.995253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:54.995220 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:43:55.002682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:55.002363 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:43:55.017866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:55.017840 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:43:55.069107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:55.069069 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:55.069352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:55.069325 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:55.210389 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:55.210304 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:55.212352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:55.212332 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:43:58.279173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.279135 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27"] Nov 03 22:43:58.280163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.280142 4898 cpu_manager.go:414] "containerMap: removing container" podUID="78156a15-e7f0-4de1-a254-20e2e4a16100" containerName="insights-gathering" Nov 03 22:43:58.280249 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.280166 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78156a15-e7f0-4de1-a254-20e2e4a16100" containerName="insights-gathering" Nov 03 22:43:58.280327 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.280316 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="78156a15-e7f0-4de1-a254-20e2e4a16100" containerName="insights-gathering" Nov 03 22:43:58.284347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.284324 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.337766 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.337738 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:43:58.337960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.337935 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:43:58.338059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.337954 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:43:58.338059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.337958 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8qdxr\"" Nov 03 22:43:58.348645 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.348615 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsxb\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-kube-api-access-9rsxb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.348776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.348664 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf8def-9505-4d56-a7b2-1f979519ebc0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.348864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.348827 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.348993 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.348967 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.349059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.349044 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.349136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.349119 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.349197 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.349183 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.391910 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.391873 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27"] Nov 03 22:43:58.450974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.450938 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.451141 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.450997 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.451141 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.451067 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsxb\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-kube-api-access-9rsxb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.451353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.451329 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf8def-9505-4d56-a7b2-1f979519ebc0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.451515 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.451496 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.451644 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.451628 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.451712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.451696 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.451810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.451764 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.453717 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.453693 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf8def-9505-4d56-a7b2-1f979519ebc0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.454530 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.454506 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.454654 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.454608 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.455165 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.455141 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.512646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.512610 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.512901 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.512878 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsxb\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-kube-api-access-9rsxb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmb27\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:58.596992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:58.596963 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:43:59.043696 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:43:59.043664 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcf8def_9505_4d56_a7b2_1f979519ebc0.slice/crio-04a4be383e727e73df14b3a65b51964212bacc9787fd49d226fae202f7489387 WatchSource:0}: Error finding container 04a4be383e727e73df14b3a65b51964212bacc9787fd49d226fae202f7489387: Status 404 returned error can't find the container with id 04a4be383e727e73df14b3a65b51964212bacc9787fd49d226fae202f7489387 Nov 03 22:43:59.044203 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:59.044174 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27"] Nov 03 22:43:59.046368 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:59.046348 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:43:59.962535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:43:59.962494 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" event={"ID":"bbcf8def-9505-4d56-a7b2-1f979519ebc0","Type":"ContainerStarted","Data":"04a4be383e727e73df14b3a65b51964212bacc9787fd49d226fae202f7489387"} Nov 03 22:44:04.521009 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:04.520974 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:04.521300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:04.521043 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:05.060636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:05.060594 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" event={"ID":"bbcf8def-9505-4d56-a7b2-1f979519ebc0","Type":"ContainerStarted","Data":"4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da"} Nov 03 22:44:05.060987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:05.060920 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:44:05.062805 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:05.062766 4898 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hmb27 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:44:05.062910 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:05.062844 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" podUID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:44:05.284716 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:05.284666 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" podStartSLOduration=1.811415122 podStartE2EDuration="7.284651046s" totalImagesPullingTime="5.473235924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:05.232441217 +0000 UTC m=+1288.445210286" watchObservedRunningTime="2026-11-03 22:44:05.284651046 +0000 UTC m=+1288.497420097" Nov 03 22:44:06.090080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:06.090042 4898 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hmb27 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:44:06.090749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:06.090721 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" podUID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:44:09.086923 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:09.086890 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:44:19.388029 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.387937 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9"] Nov 03 22:44:19.406366 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.397725 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520008 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ngt\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-kube-api-access-z2ngt\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520086 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520153 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520184 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520257 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520280 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520323 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520401 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.522887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.520460 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.577743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.577490 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9"] Nov 03 22:44:19.622939 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.622900 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.623182 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.623162 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.623333 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.623315 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.623491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.623461 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.623607 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.623495 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.623767 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.623749 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.624588 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.623751 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.624729 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.624400 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.624877 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.624646 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ngt\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-kube-api-access-z2ngt\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.625035 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.625017 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.625119 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.625096 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.625178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.625133 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.625517 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.625499 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.625628 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.625514 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.628157 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.628134 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.628733 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.628713 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.691033 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.687969 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ngt\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-kube-api-access-z2ngt\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.692905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.691812 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:19.838732 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:44:19.838633 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:44:19.917060 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:19.917027 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:20.413127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:20.413100 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9"] Nov 03 22:44:20.414324 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:44:20.414297 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eeb37c6_d4ef_46bc_b61f_cef76ff7fd20.slice/crio-4dbed896cffb6d2daf047b7c4dcd5a3c5f64bdec8abe07cbc8e1da8a86b02891 WatchSource:0}: Error finding container 4dbed896cffb6d2daf047b7c4dcd5a3c5f64bdec8abe07cbc8e1da8a86b02891: Status 404 returned error can't find the container with id 4dbed896cffb6d2daf047b7c4dcd5a3c5f64bdec8abe07cbc8e1da8a86b02891 Nov 03 22:44:20.416845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:20.416813 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:20.416911 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:20.416879 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:20.416948 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:20.416909 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:21.313898 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:21.313863 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" event={"ID":"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20","Type":"ContainerStarted","Data":"3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924"} Nov 03 22:44:21.314096 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:21.313924 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" event={"ID":"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20","Type":"ContainerStarted","Data":"4dbed896cffb6d2daf047b7c4dcd5a3c5f64bdec8abe07cbc8e1da8a86b02891"} Nov 03 22:44:21.560813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:21.540358 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podStartSLOduration=2.540333654 podStartE2EDuration="2.540333654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:21.487366118 +0000 UTC m=+1304.700135169" watchObservedRunningTime="2026-11-03 22:44:21.540333654 +0000 UTC m=+1304.753102705" Nov 03 22:44:21.917260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:21.917224 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:21.919729 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:21.919689 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:21.919926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:21.919776 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:22.918203 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:22.918165 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:22.918587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:22.918244 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:23.917667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:23.917632 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:23.917864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:23.917688 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:24.930001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:24.929960 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:24.930621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:24.930585 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:25.917920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:25.917883 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:25.918103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:25.917955 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:26.920289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:26.919849 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:26.920289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:26.919917 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:27.403329 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:27.403289 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:27.403621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:27.403578 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.001833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:27.999871 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.024064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:28.023208 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:28.024064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:28.023270 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:28.024064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:28.023350 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.379814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:28.376295 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.447815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:28.447107 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:44:29.036812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.035694 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:44:29.036812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.035773 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 22:44:29.036812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.036677 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:44:29.049071 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.037608 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.381069 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.381010 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.492820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.491877 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.638491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.638387 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:44:29.638768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.638750 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:44:29.638958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.638944 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:44:29.639550 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.639534 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:44:29.639694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.639684 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:44:29.942864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:29.932310 4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:30.049880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:30.039471 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:30.049880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:30.039719 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:30.524813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:30.512666 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:44:31.051811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.044728 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.104814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.047876 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 22:44:31.133833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.132451 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:31.133833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.132519 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.416709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.415093 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.513273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.513225 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 22:44:31.565814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.545850 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:44:31.565814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.555874 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:44:31.654999 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.651070 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:44:31.822265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.822098 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:31.824053 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.824029 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:31.832087 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:44:31.832053 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:44:49.836200 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:44:49.836156 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:45:00.206491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.206454 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr"] Nov 03 22:45:00.212594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.212562 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.213155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.213133 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6"] Nov 03 22:45:00.219191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.219151 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx"] Nov 03 22:45:00.219830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.219772 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.224745 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.224723 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:00.316232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.316164 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr"] Nov 03 22:45:00.384817 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.367481 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:45:00.454812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.436002 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ksz\" (UniqueName: \"kubernetes.io/projected/a6cdf855-ea27-4294-8f79-ab3a30589386-kube-api-access-q6ksz\") pod \"maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.455354 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.455263 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cdf855-ea27-4294-8f79-ab3a30589386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.455589 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.455567 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6653d7-4f7c-4f1b-852e-bec023b0968f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.455859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.455840 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g58bt\" (UniqueName: \"kubernetes.io/projected/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-kube-api-access-g58bt\") pod \"maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:00.456098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.456077 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:00.463375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.463306 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqll4\" (UniqueName: \"kubernetes.io/projected/4b6653d7-4f7c-4f1b-852e-bec023b0968f-kube-api-access-pqll4\") pod \"maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.568843 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cdf855-ea27-4294-8f79-ab3a30589386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.568920 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6653d7-4f7c-4f1b-852e-bec023b0968f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.568995 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g58bt\" (UniqueName: \"kubernetes.io/projected/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-kube-api-access-g58bt\") pod \"maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.569032 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.569078 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqll4\" (UniqueName: \"kubernetes.io/projected/4b6653d7-4f7c-4f1b-852e-bec023b0968f-kube-api-access-pqll4\") pod \"maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.569171 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ksz\" (UniqueName: \"kubernetes.io/projected/a6cdf855-ea27-4294-8f79-ab3a30589386-kube-api-access-q6ksz\") pod \"maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.570501 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cdf855-ea27-4294-8f79-ab3a30589386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.571129 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6653d7-4f7c-4f1b-852e-bec023b0968f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.572043 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:00.602883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.597366 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6"] Nov 03 22:45:00.608950 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.608860 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx"] Nov 03 22:45:00.633896 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.632726 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ksz\" (UniqueName: \"kubernetes.io/projected/a6cdf855-ea27-4294-8f79-ab3a30589386-kube-api-access-q6ksz\") pod \"maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.635986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.635904 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g58bt\" (UniqueName: \"kubernetes.io/projected/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-kube-api-access-g58bt\") pod \"maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:00.638713 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.638684 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqll4\" (UniqueName: \"kubernetes.io/projected/4b6653d7-4f7c-4f1b-852e-bec023b0968f-kube-api-access-pqll4\") pod \"maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.832496 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.832458 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:00.841103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.841071 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:00.853614 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:00.853581 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:01.295537 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:45:01.295481 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6653d7_4f7c_4f1b_852e_bec023b0968f.slice/crio-0bedf085a04dfeda1fe2d5554dd6630b1d113d9823bd7ed29e65b34da36cdcde WatchSource:0}: Error finding container 0bedf085a04dfeda1fe2d5554dd6630b1d113d9823bd7ed29e65b34da36cdcde: Status 404 returned error can't find the container with id 0bedf085a04dfeda1fe2d5554dd6630b1d113d9823bd7ed29e65b34da36cdcde Nov 03 22:45:01.297522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:01.297494 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6"] Nov 03 22:45:01.320244 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:01.320210 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr"] Nov 03 22:45:01.395095 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:01.394633 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx"] Nov 03 22:45:02.052813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:02.051533 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" event={"ID":"a6cdf855-ea27-4294-8f79-ab3a30589386","Type":"ContainerStarted","Data":"d700ac50bbd4fd4c728f387ac5e08ef5b9cd2333c39e299fd2b64a43f78cb720"} Nov 03 22:45:02.071975 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:02.071855 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" event={"ID":"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a","Type":"ContainerStarted","Data":"8ac83764a272fe7abbe12f191566b3adb76c9f6d4d515d257b8eb4929f19de98"} Nov 03 22:45:02.074087 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:02.074006 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" event={"ID":"4b6653d7-4f7c-4f1b-852e-bec023b0968f","Type":"ContainerStarted","Data":"0bedf085a04dfeda1fe2d5554dd6630b1d113d9823bd7ed29e65b34da36cdcde"} Nov 03 22:45:03.106044 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.105999 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r"] Nov 03 22:45:03.145043 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.145005 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r"] Nov 03 22:45:03.145385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.145363 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.193872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.192807 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.193872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.192951 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnwm\" (UniqueName: \"kubernetes.io/projected/fff4c634-689d-4147-bcff-a9f548b6a65d-kube-api-access-gwnwm\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.193872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.193019 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff4c634-689d-4147-bcff-a9f548b6a65d-openshift-service-ca\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.193872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.193105 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-metrics-certs\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.266845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.263826 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c6905007-serving-cert\"" Nov 03 22:45:03.344433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.300232 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.344433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.300373 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnwm\" (UniqueName: \"kubernetes.io/projected/fff4c634-689d-4147-bcff-a9f548b6a65d-kube-api-access-gwnwm\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.344433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.300450 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff4c634-689d-4147-bcff-a9f548b6a65d-openshift-service-ca\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.344433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.300536 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-metrics-certs\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.344433 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:03.301910 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-serving-cert: secret "maas-api-e2e-ait-c6905007-serving-cert" not found Nov 03 22:45:03.344433 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:03.301994 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls podName:fff4c634-689d-4147-bcff-a9f548b6a65d nodeName:}" failed. No retries permitted until 2026-11-03 22:45:03.801970553 +0000 UTC m=+1347.014739596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls") pod "maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" (UID: "fff4c634-689d-4147-bcff-a9f548b6a65d") : secret "maas-api-e2e-ait-c6905007-serving-cert" not found Nov 03 22:45:03.344433 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.306987 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff4c634-689d-4147-bcff-a9f548b6a65d-openshift-service-ca\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.392288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.392126 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 22:45:03.557832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.550434 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-metrics-certs\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.557832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.550699 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.557832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.552405 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.716677 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.716555 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnwm\" (UniqueName: \"kubernetes.io/projected/fff4c634-689d-4147-bcff-a9f548b6a65d-kube-api-access-gwnwm\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.854973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:03.854938 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:03.871333 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:03.871303 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-serving-cert: secret "maas-api-e2e-ait-c6905007-serving-cert" not found Nov 03 22:45:03.871780 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:03.871763 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls podName:fff4c634-689d-4147-bcff-a9f548b6a65d nodeName:}" failed. No retries permitted until 2026-11-03 22:45:04.871649465 +0000 UTC m=+1348.084418514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls") pod "maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" (UID: "fff4c634-689d-4147-bcff-a9f548b6a65d") : secret "maas-api-e2e-ait-c6905007-serving-cert" not found Nov 03 22:45:04.061353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.061246 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.412813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.399916 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.412813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.401815 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:45:04.558807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.556916 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:45:04.558807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.557759 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:45:04.569843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.559145 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:45:04.569843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.559210 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:45:05.115839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:04.951834 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:05.269814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:05.263018 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 22:45:05.425859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:05.408487 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.425859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:05.417421 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\") pod \"maas-api-e2e-ait-c6905007-65798b8d5-bdf8r\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:05.483821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:05.478849 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:05.503824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:05.498009 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 22:45:05.516305 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:05.510542 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 22:45:06.246117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:06.232314 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r"] Nov 03 22:45:06.256807 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:45:06.249775 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff4c634_689d_4147_bcff_a9f548b6a65d.slice/crio-2f3762f5b4c20be8d97c6e26660d0604ff11781870e2fc28e7eed6c7e0b8aa7d WatchSource:0}: Error finding container 2f3762f5b4c20be8d97c6e26660d0604ff11781870e2fc28e7eed6c7e0b8aa7d: Status 404 returned error can't find the container with id 2f3762f5b4c20be8d97c6e26660d0604ff11781870e2fc28e7eed6c7e0b8aa7d Nov 03 22:45:06.499364 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:06.499272 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" event={"ID":"fff4c634-689d-4147-bcff-a9f548b6a65d","Type":"ContainerStarted","Data":"2f3762f5b4c20be8d97c6e26660d0604ff11781870e2fc28e7eed6c7e0b8aa7d"} Nov 03 22:45:07.528132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:07.528068 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" event={"ID":"fff4c634-689d-4147-bcff-a9f548b6a65d","Type":"ContainerStarted","Data":"07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b"} Nov 03 22:45:07.530360 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:07.530327 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:07.669154 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:07.669097 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" podStartSLOduration=3.642832959 podStartE2EDuration="4.669078134s" totalImagesPullingTime="1.026245175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:07.664315984 +0000 UTC m=+1350.877085036" watchObservedRunningTime="2026-11-03 22:45:07.669078134 +0000 UTC m=+1350.881847239" Nov 03 22:45:11.584760 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.584716 4898 generic.go:356] "Generic (PLEG): container finished" podID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerID="d2d5e660c2f9a9a2a7e42cb9b12154f181678054fdda8b6cd713be9323080c35" exitCode=60 Nov 03 22:45:11.585291 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.584845 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" event={"ID":"4b6653d7-4f7c-4f1b-852e-bec023b0968f","Type":"ContainerDied","Data":"d2d5e660c2f9a9a2a7e42cb9b12154f181678054fdda8b6cd713be9323080c35"} Nov 03 22:45:11.585291 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.585184 4898 scope.go:122] "RemoveContainer" containerID="d2d5e660c2f9a9a2a7e42cb9b12154f181678054fdda8b6cd713be9323080c35" Nov 03 22:45:11.587584 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.586820 4898 generic.go:356] "Generic (PLEG): container finished" podID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerID="518c05ea952db0957e066cd339e37a50cd80368b123988ff5a29d98bec6de6da" exitCode=60 Nov 03 22:45:11.587584 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.586925 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" event={"ID":"a6cdf855-ea27-4294-8f79-ab3a30589386","Type":"ContainerDied","Data":"518c05ea952db0957e066cd339e37a50cd80368b123988ff5a29d98bec6de6da"} Nov 03 22:45:11.587584 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.587198 4898 scope.go:122] "RemoveContainer" containerID="518c05ea952db0957e066cd339e37a50cd80368b123988ff5a29d98bec6de6da" Nov 03 22:45:11.589015 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.588991 4898 generic.go:356] "Generic (PLEG): container finished" podID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerID="8cabd0508a06079fe16503049e8396164442b3698bfd385ff63dabebaf9a2691" exitCode=60 Nov 03 22:45:11.589144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.589077 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" event={"ID":"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a","Type":"ContainerDied","Data":"8cabd0508a06079fe16503049e8396164442b3698bfd385ff63dabebaf9a2691"} Nov 03 22:45:11.589332 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:11.589319 4898 scope.go:122] "RemoveContainer" containerID="8cabd0508a06079fe16503049e8396164442b3698bfd385ff63dabebaf9a2691" Nov 03 22:45:12.597900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:12.597864 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" event={"ID":"4b6653d7-4f7c-4f1b-852e-bec023b0968f","Type":"ContainerStarted","Data":"0b5b37c639ba1d97137dfe96cf868d5d6c5ad87b8e0ebbcfa9b94cdad52555ea"} Nov 03 22:45:12.600019 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:12.599991 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" event={"ID":"a6cdf855-ea27-4294-8f79-ab3a30589386","Type":"ContainerStarted","Data":"06ca58ff5638eacc226aefdf41b863c0f812fecaa6114c3dfde96c1f16df2bbc"} Nov 03 22:45:12.602159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:12.602134 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" event={"ID":"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a","Type":"ContainerStarted","Data":"36c57651475c56aafea89ccf7c16c7f44e0059acb7299fb9a966645d7c282b17"} Nov 03 22:45:12.715980 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:12.715929 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" podStartSLOduration=3.909462054 podStartE2EDuration="12.715915325s" totalImagesPullingTime="8.806453271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:12.713485895 +0000 UTC m=+1355.926254965" watchObservedRunningTime="2026-11-03 22:45:12.715915325 +0000 UTC m=+1355.928684389" Nov 03 22:45:12.852012 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:12.851890 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" podStartSLOduration=3.7699693659999998 podStartE2EDuration="12.85185994s" totalImagesPullingTime="9.081890574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:12.830267418 +0000 UTC m=+1356.043036489" watchObservedRunningTime="2026-11-03 22:45:12.85185994 +0000 UTC m=+1356.064629120" Nov 03 22:45:12.949294 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:12.949229 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" podStartSLOduration=4.255359221 podStartE2EDuration="12.94920893s" totalImagesPullingTime="8.693849709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:12.946928574 +0000 UTC m=+1356.159697643" watchObservedRunningTime="2026-11-03 22:45:12.94920893 +0000 UTC m=+1356.161977984" Nov 03 22:45:13.613285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.613248 4898 generic.go:356] "Generic (PLEG): container finished" podID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerID="06ca58ff5638eacc226aefdf41b863c0f812fecaa6114c3dfde96c1f16df2bbc" exitCode=60 Nov 03 22:45:13.613781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.613342 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" event={"ID":"a6cdf855-ea27-4294-8f79-ab3a30589386","Type":"ContainerDied","Data":"06ca58ff5638eacc226aefdf41b863c0f812fecaa6114c3dfde96c1f16df2bbc"} Nov 03 22:45:13.613781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.613426 4898 scope.go:122] "RemoveContainer" containerID="518c05ea952db0957e066cd339e37a50cd80368b123988ff5a29d98bec6de6da" Nov 03 22:45:13.613781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.613668 4898 scope.go:122] "RemoveContainer" containerID="06ca58ff5638eacc226aefdf41b863c0f812fecaa6114c3dfde96c1f16df2bbc" Nov 03 22:45:13.614083 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:13.614055 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr_odh-ai-gateway-infra(a6cdf855-ea27-4294-8f79-ab3a30589386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" Nov 03 22:45:13.615731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.615709 4898 generic.go:356] "Generic (PLEG): container finished" podID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerID="36c57651475c56aafea89ccf7c16c7f44e0059acb7299fb9a966645d7c282b17" exitCode=60 Nov 03 22:45:13.615880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.615812 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" event={"ID":"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a","Type":"ContainerDied","Data":"36c57651475c56aafea89ccf7c16c7f44e0059acb7299fb9a966645d7c282b17"} Nov 03 22:45:13.616435 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.616419 4898 scope.go:122] "RemoveContainer" containerID="36c57651475c56aafea89ccf7c16c7f44e0059acb7299fb9a966645d7c282b17" Nov 03 22:45:13.616828 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:13.616769 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx_odh-ai-gateway-infra(e06d0b27-6f23-4ea3-9640-c3f0468e9a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" Nov 03 22:45:13.618002 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.617982 4898 generic.go:356] "Generic (PLEG): container finished" podID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerID="0b5b37c639ba1d97137dfe96cf868d5d6c5ad87b8e0ebbcfa9b94cdad52555ea" exitCode=60 Nov 03 22:45:13.618081 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.618045 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" event={"ID":"4b6653d7-4f7c-4f1b-852e-bec023b0968f","Type":"ContainerDied","Data":"0b5b37c639ba1d97137dfe96cf868d5d6c5ad87b8e0ebbcfa9b94cdad52555ea"} Nov 03 22:45:13.618285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.618272 4898 scope.go:122] "RemoveContainer" containerID="0b5b37c639ba1d97137dfe96cf868d5d6c5ad87b8e0ebbcfa9b94cdad52555ea" Nov 03 22:45:13.618527 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:13.618511 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6_odh-ai-gateway-infra(4b6653d7-4f7c-4f1b-852e-bec023b0968f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" Nov 03 22:45:13.636640 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.636618 4898 scope.go:122] "RemoveContainer" containerID="8cabd0508a06079fe16503049e8396164442b3698bfd385ff63dabebaf9a2691" Nov 03 22:45:13.670697 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:13.670673 4898 scope.go:122] "RemoveContainer" containerID="d2d5e660c2f9a9a2a7e42cb9b12154f181678054fdda8b6cd713be9323080c35" Nov 03 22:45:14.574757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.574730 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:45:14.626465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.626434 4898 scope.go:122] "RemoveContainer" containerID="0b5b37c639ba1d97137dfe96cf868d5d6c5ad87b8e0ebbcfa9b94cdad52555ea" Nov 03 22:45:14.626973 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:14.626848 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6_odh-ai-gateway-infra(4b6653d7-4f7c-4f1b-852e-bec023b0968f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" Nov 03 22:45:14.628265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.628228 4898 scope.go:122] "RemoveContainer" containerID="06ca58ff5638eacc226aefdf41b863c0f812fecaa6114c3dfde96c1f16df2bbc" Nov 03 22:45:14.628573 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:14.628551 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr_odh-ai-gateway-infra(a6cdf855-ea27-4294-8f79-ab3a30589386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" Nov 03 22:45:14.629804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.629775 4898 scope.go:122] "RemoveContainer" containerID="36c57651475c56aafea89ccf7c16c7f44e0059acb7299fb9a966645d7c282b17" Nov 03 22:45:14.630098 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:14.630078 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx_odh-ai-gateway-infra(e06d0b27-6f23-4ea3-9640-c3f0468e9a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" Nov 03 22:45:14.631142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.631117 4898 generic.go:356] "Generic (PLEG): container finished" podID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerID="faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2" exitCode=0 Nov 03 22:45:14.631240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.631181 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" event={"ID":"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8","Type":"ContainerDied","Data":"faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2"} Nov 03 22:45:14.631511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.631498 4898 scope.go:122] "RemoveContainer" containerID="faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2" Nov 03 22:45:14.641839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.636821 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:14.641839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.636912 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:14.641839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:14.636957 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:15.642741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:15.642701 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" event={"ID":"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8","Type":"ContainerStarted","Data":"769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5"} Nov 03 22:45:16.295652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:16.295616 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:45:16.296681 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:16.296653 4898 patch_prober.go:29] interesting pod/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:45:16.296827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:16.296705 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 22:45:16.327070 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:16.327037 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:45:17.295427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:17.295393 4898 patch_prober.go:29] interesting pod/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:45:17.295820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:17.295446 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 22:45:18.295541 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:18.295507 4898 patch_prober.go:29] interesting pod/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:45:18.296001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:18.295560 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 22:45:19.295108 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:19.295072 4898 patch_prober.go:29] interesting pod/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:45:19.295293 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:19.295133 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 22:45:19.890291 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:19.890247 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:45:20.295354 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:20.295248 4898 patch_prober.go:29] interesting pod/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:45:20.295354 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:20.295316 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 22:45:21.295294 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:21.295257 4898 patch_prober.go:29] interesting pod/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:45:21.295767 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:21.295326 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 22:45:22.300128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:22.300091 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:45:22.301210 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:22.301187 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:45:25.899716 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:25.899682 4898 scope.go:122] "RemoveContainer" containerID="0b5b37c639ba1d97137dfe96cf868d5d6c5ad87b8e0ebbcfa9b94cdad52555ea" Nov 03 22:45:25.900191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:25.899877 4898 scope.go:122] "RemoveContainer" containerID="36c57651475c56aafea89ccf7c16c7f44e0059acb7299fb9a966645d7c282b17" Nov 03 22:45:25.900191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:25.900161 4898 scope.go:122] "RemoveContainer" containerID="06ca58ff5638eacc226aefdf41b863c0f812fecaa6114c3dfde96c1f16df2bbc" Nov 03 22:45:26.770903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:26.770756 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" event={"ID":"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a","Type":"ContainerStarted","Data":"abe460999c6781f5e7c071968e84b63426bd4a72a3e93de9c9a5211f001b4a9d"} Nov 03 22:45:26.784820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:26.782814 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" event={"ID":"4b6653d7-4f7c-4f1b-852e-bec023b0968f","Type":"ContainerStarted","Data":"7e1c27458b0b44efe982b27223209fd3ab2fde03adf2474045baeebad1b3802f"} Nov 03 22:45:26.789350 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:26.786647 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" event={"ID":"a6cdf855-ea27-4294-8f79-ab3a30589386","Type":"ContainerStarted","Data":"caa79e1059d2112ef796d8154a381132096fd13174c0433e87b75b7092e0e3c8"} Nov 03 22:45:27.795343 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.795297 4898 generic.go:356] "Generic (PLEG): container finished" podID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerID="caa79e1059d2112ef796d8154a381132096fd13174c0433e87b75b7092e0e3c8" exitCode=60 Nov 03 22:45:27.795846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.795375 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" event={"ID":"a6cdf855-ea27-4294-8f79-ab3a30589386","Type":"ContainerDied","Data":"caa79e1059d2112ef796d8154a381132096fd13174c0433e87b75b7092e0e3c8"} Nov 03 22:45:27.795846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.795480 4898 scope.go:122] "RemoveContainer" containerID="06ca58ff5638eacc226aefdf41b863c0f812fecaa6114c3dfde96c1f16df2bbc" Nov 03 22:45:27.796010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.795908 4898 scope.go:122] "RemoveContainer" containerID="caa79e1059d2112ef796d8154a381132096fd13174c0433e87b75b7092e0e3c8" Nov 03 22:45:27.796274 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:27.796248 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr_odh-ai-gateway-infra(a6cdf855-ea27-4294-8f79-ab3a30589386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" Nov 03 22:45:27.797480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.797460 4898 generic.go:356] "Generic (PLEG): container finished" podID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerID="abe460999c6781f5e7c071968e84b63426bd4a72a3e93de9c9a5211f001b4a9d" exitCode=60 Nov 03 22:45:27.797578 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.797530 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" event={"ID":"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a","Type":"ContainerDied","Data":"abe460999c6781f5e7c071968e84b63426bd4a72a3e93de9c9a5211f001b4a9d"} Nov 03 22:45:27.797908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.797893 4898 scope.go:122] "RemoveContainer" containerID="abe460999c6781f5e7c071968e84b63426bd4a72a3e93de9c9a5211f001b4a9d" Nov 03 22:45:27.798236 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:27.798214 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx_odh-ai-gateway-infra(e06d0b27-6f23-4ea3-9640-c3f0468e9a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" Nov 03 22:45:27.799445 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.799419 4898 generic.go:356] "Generic (PLEG): container finished" podID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerID="7e1c27458b0b44efe982b27223209fd3ab2fde03adf2474045baeebad1b3802f" exitCode=60 Nov 03 22:45:27.799535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.799486 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" event={"ID":"4b6653d7-4f7c-4f1b-852e-bec023b0968f","Type":"ContainerDied","Data":"7e1c27458b0b44efe982b27223209fd3ab2fde03adf2474045baeebad1b3802f"} Nov 03 22:45:27.799755 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.799741 4898 scope.go:122] "RemoveContainer" containerID="7e1c27458b0b44efe982b27223209fd3ab2fde03adf2474045baeebad1b3802f" Nov 03 22:45:27.799978 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:27.799961 4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6_odh-ai-gateway-infra(4b6653d7-4f7c-4f1b-852e-bec023b0968f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" Nov 03 22:45:27.807947 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.807929 4898 scope.go:122] "RemoveContainer" containerID="36c57651475c56aafea89ccf7c16c7f44e0059acb7299fb9a966645d7c282b17" Nov 03 22:45:27.823206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.823185 4898 scope.go:122] "RemoveContainer" containerID="0b5b37c639ba1d97137dfe96cf868d5d6c5ad87b8e0ebbcfa9b94cdad52555ea" Nov 03 22:45:27.903218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:27.903187 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx"] Nov 03 22:45:28.014365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:28.014326 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr"] Nov 03 22:45:28.144042 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:28.144001 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6"] Nov 03 22:45:29.016839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.009329 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:29.092542 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.092518 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:29.095988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.095967 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:29.102773 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.102743 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6653d7-4f7c-4f1b-852e-bec023b0968f-kube-api-access-pqll4\" (UniqueName: \"kubernetes.io/projected/4b6653d7-4f7c-4f1b-852e-bec023b0968f-kube-api-access-pqll4\") pod \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " Nov 03 22:45:29.102908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.102830 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6653d7-4f7c-4f1b-852e-bec023b0968f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6653d7-4f7c-4f1b-852e-bec023b0968f-maas-api-service-ca\") pod \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\" (UID: \"4b6653d7-4f7c-4f1b-852e-bec023b0968f\") " Nov 03 22:45:29.103174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.103148 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6653d7-4f7c-4f1b-852e-bec023b0968f-maas-api-service-ca" pod "4b6653d7-4f7c-4f1b-852e-bec023b0968f" (UID: "4b6653d7-4f7c-4f1b-852e-bec023b0968f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:29.103545 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.103526 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6653d7-4f7c-4f1b-852e-bec023b0968f-maas-api-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:29.104928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.104907 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6653d7-4f7c-4f1b-852e-bec023b0968f-kube-api-access-pqll4" pod "4b6653d7-4f7c-4f1b-852e-bec023b0968f" (UID: "4b6653d7-4f7c-4f1b-852e-bec023b0968f"). InnerVolumeSpecName "kube-api-access-pqll4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:29.204545 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.204510 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6cdf855-ea27-4294-8f79-ab3a30589386-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cdf855-ea27-4294-8f79-ab3a30589386-maas-api-service-ca\") pod \"a6cdf855-ea27-4294-8f79-ab3a30589386\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " Nov 03 22:45:29.204847 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.204820 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-kube-api-access-g58bt\" (UniqueName: \"kubernetes.io/projected/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-kube-api-access-g58bt\") pod \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " Nov 03 22:45:29.205026 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.204939 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-maas-api-service-ca\") pod \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\" (UID: \"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a\") " Nov 03 22:45:29.205026 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.204945 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6cdf855-ea27-4294-8f79-ab3a30589386-maas-api-service-ca" pod "a6cdf855-ea27-4294-8f79-ab3a30589386" (UID: "a6cdf855-ea27-4294-8f79-ab3a30589386"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:29.205378 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.205351 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-maas-api-service-ca" pod "e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" (UID: "e06d0b27-6f23-4ea3-9640-c3f0468e9a2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:29.205523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.205505 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6cdf855-ea27-4294-8f79-ab3a30589386-kube-api-access-q6ksz\" (UniqueName: \"kubernetes.io/projected/a6cdf855-ea27-4294-8f79-ab3a30589386-kube-api-access-q6ksz\") pod \"a6cdf855-ea27-4294-8f79-ab3a30589386\" (UID: \"a6cdf855-ea27-4294-8f79-ab3a30589386\") " Nov 03 22:45:29.206244 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.206221 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cdf855-ea27-4294-8f79-ab3a30589386-maas-api-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:29.206360 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.206247 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqll4\" (UniqueName: \"kubernetes.io/projected/4b6653d7-4f7c-4f1b-852e-bec023b0968f-kube-api-access-pqll4\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:29.206360 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.206262 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-maas-api-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:29.207270 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.207249 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-kube-api-access-g58bt" pod "e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" (UID: "e06d0b27-6f23-4ea3-9640-c3f0468e9a2a"). InnerVolumeSpecName "kube-api-access-g58bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:29.207962 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.207938 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6cdf855-ea27-4294-8f79-ab3a30589386-kube-api-access-q6ksz" pod "a6cdf855-ea27-4294-8f79-ab3a30589386" (UID: "a6cdf855-ea27-4294-8f79-ab3a30589386"). InnerVolumeSpecName "kube-api-access-q6ksz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:29.307656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.307623 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g58bt\" (UniqueName: \"kubernetes.io/projected/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a-kube-api-access-g58bt\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:29.307656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.307649 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ksz\" (UniqueName: \"kubernetes.io/projected/a6cdf855-ea27-4294-8f79-ab3a30589386-kube-api-access-q6ksz\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:29.825907 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.825858 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" event={"ID":"e06d0b27-6f23-4ea3-9640-c3f0468e9a2a","Type":"ContainerDied","Data":"8ac83764a272fe7abbe12f191566b3adb76c9f6d4d515d257b8eb4929f19de98"} Nov 03 22:45:29.826105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.825931 4898 scope.go:122] "RemoveContainer" containerID="abe460999c6781f5e7c071968e84b63426bd4a72a3e93de9c9a5211f001b4a9d" Nov 03 22:45:29.826105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.826041 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx" Nov 03 22:45:29.829535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.829490 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" event={"ID":"4b6653d7-4f7c-4f1b-852e-bec023b0968f","Type":"ContainerDied","Data":"0bedf085a04dfeda1fe2d5554dd6630b1d113d9823bd7ed29e65b34da36cdcde"} Nov 03 22:45:29.829667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.829596 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6" Nov 03 22:45:29.836559 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.836530 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" event={"ID":"a6cdf855-ea27-4294-8f79-ab3a30589386","Type":"ContainerDied","Data":"d700ac50bbd4fd4c728f387ac5e08ef5b9cd2333c39e299fd2b64a43f78cb720"} Nov 03 22:45:29.836684 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.836628 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr" Nov 03 22:45:29.842418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.841977 4898 scope.go:122] "RemoveContainer" containerID="7e1c27458b0b44efe982b27223209fd3ab2fde03adf2474045baeebad1b3802f" Nov 03 22:45:29.856004 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:29.855982 4898 scope.go:122] "RemoveContainer" containerID="caa79e1059d2112ef796d8154a381132096fd13174c0433e87b75b7092e0e3c8" Nov 03 22:45:30.030658 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.030618 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6"] Nov 03 22:45:30.038860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.038832 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5e72ad-29895765-b4cm6"] Nov 03 22:45:30.157025 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.154502 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx"] Nov 03 22:45:30.178722 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.178673 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-027a6fbe-29895765-sg8wx"] Nov 03 22:45:30.268269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.268233 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr"] Nov 03 22:45:30.273845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.273817 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c082ab23-29895765-rsprr"] Nov 03 22:45:30.908272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.908232 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" path="/var/lib/kubelet/pods/4b6653d7-4f7c-4f1b-852e-bec023b0968f/volumes" Nov 03 22:45:30.908901 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.908880 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" path="/var/lib/kubelet/pods/a6cdf855-ea27-4294-8f79-ab3a30589386/volumes" Nov 03 22:45:30.909454 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:45:30.909435 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" path="/var/lib/kubelet/pods/e06d0b27-6f23-4ea3-9640-c3f0468e9a2a/volumes" Nov 03 22:45:49.836206 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:45:49.836161 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:46:18.030811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.029972 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt"] Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.030909 4898 cpu_manager.go:414] "containerMap: removing container" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.030929 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.030941 4898 cpu_manager.go:414] "containerMap: removing container" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.030948 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.030991 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.030998 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031012 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031018 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031028 4898 cpu_manager.go:414] "containerMap: removing container" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031034 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031049 4898 cpu_manager.go:414] "containerMap: removing container" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031055 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031085 4898 cpu_manager.go:414] "containerMap: removing container" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031091 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031251 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031264 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031273 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031285 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031296 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031305 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:46:18.031420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.031314 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:46:18.395310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.395277 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt"] Nov 03 22:46:18.395651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.395632 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.491407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.491369 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.491698 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.491677 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c9ca07-ee0b-474f-9954-087baaf18359-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.491916 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.491898 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6wxk\" (UniqueName: \"kubernetes.io/projected/79c9ca07-ee0b-474f-9954-087baaf18359-kube-api-access-g6wxk\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.492148 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.492129 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.500201 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.500131 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-65cac0-serving-cert\"" Nov 03 22:46:18.594001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.593958 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.594222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.594171 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.594222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.594206 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c9ca07-ee0b-474f-9954-087baaf18359-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.594348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.594270 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wxk\" (UniqueName: \"kubernetes.io/projected/79c9ca07-ee0b-474f-9954-087baaf18359-kube-api-access-g6wxk\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.595238 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.595209 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c9ca07-ee0b-474f-9954-087baaf18359-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.598299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.597175 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.598299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.597860 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.669066 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.668985 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wxk\" (UniqueName: \"kubernetes.io/projected/79c9ca07-ee0b-474f-9954-087baaf18359-kube-api-access-g6wxk\") pod \"maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:18.740629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:18.740595 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:19.189158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:19.189128 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt"] Nov 03 22:46:19.190100 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:46:19.190071 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c9ca07_ee0b_474f_9954_087baaf18359.slice/crio-9db7ff2eb7589d2df63e86629cdb000b84a0c59fc9afd983abdfbbff31cbcbfa WatchSource:0}: Error finding container 9db7ff2eb7589d2df63e86629cdb000b84a0c59fc9afd983abdfbbff31cbcbfa: Status 404 returned error can't find the container with id 9db7ff2eb7589d2df63e86629cdb000b84a0c59fc9afd983abdfbbff31cbcbfa Nov 03 22:46:19.468504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:19.468418 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" event={"ID":"79c9ca07-ee0b-474f-9954-087baaf18359","Type":"ContainerStarted","Data":"9db7ff2eb7589d2df63e86629cdb000b84a0c59fc9afd983abdfbbff31cbcbfa"} Nov 03 22:46:19.845550 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:46:19.845504 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:46:20.484936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:20.483647 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" event={"ID":"79c9ca07-ee0b-474f-9954-087baaf18359","Type":"ContainerStarted","Data":"f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf"} Nov 03 22:46:20.484936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:20.484836 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:20.603987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:20.603932 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" podStartSLOduration=2.686814138 podStartE2EDuration="3.603913911s" totalImagesPullingTime="917.099773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:20.602658965 +0000 UTC m=+1423.815428043" watchObservedRunningTime="2026-11-03 22:46:20.603913911 +0000 UTC m=+1423.816682962" Nov 03 22:46:25.803827 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:25.803774 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4"] Nov 03 22:46:25.804749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:25.804713 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" containerID="cri-o://769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5" gracePeriod=30 Nov 03 22:46:27.530266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:27.530239 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 22:46:31.104013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.103988 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:46:31.208235 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208205 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-certs\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208278 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-socket\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208313 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-token\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-token\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208346 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-data\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208363 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istiod-ca-cert\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208450 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-envoy\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208495 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-kube-api-access-7nngx\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-kube-api-access-7nngx\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208491 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-certs" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:31.208662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208600 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-socket" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:31.208662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208633 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-podinfo\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208695 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-credential-socket\") pod \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\" (UID: \"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8\") " Nov 03 22:46:31.208924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208728 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-data" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:31.208924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208861 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istiod-ca-cert" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:31.209072 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.208992 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-credential-socket" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:31.209443 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.209421 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.209551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.209444 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.209551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.209458 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.209551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.209466 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.209551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.209474 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.210859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.210820 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-envoy" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:31.210979 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.210872 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-token" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:31.211161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.211131 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-podinfo" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:46:31.211209 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.211191 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-kube-api-access-7nngx" pod "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" (UID: "58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8"). InnerVolumeSpecName "kube-api-access-7nngx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:31.311348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.311312 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.311348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.311347 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nngx\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-kube-api-access-7nngx\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.311348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.311365 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.311658 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.311378 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:31.743485 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.743448 4898 generic.go:356] "Generic (PLEG): container finished" podID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerID="769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5" exitCode=0 Nov 03 22:46:31.743754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.743496 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" Nov 03 22:46:31.743754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.743525 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" event={"ID":"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8","Type":"ContainerDied","Data":"769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5"} Nov 03 22:46:31.743754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.743582 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4" event={"ID":"58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8","Type":"ContainerDied","Data":"40f5226a9dc5d99739d93f016192ba3ce93b44c01a3d266a217c58a152cba96a"} Nov 03 22:46:31.743754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.743606 4898 scope.go:122] "RemoveContainer" containerID="769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5" Nov 03 22:46:31.754642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.754617 4898 scope.go:122] "RemoveContainer" containerID="faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2" Nov 03 22:46:31.763826 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.763810 4898 scope.go:122] "RemoveContainer" containerID="769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5" Nov 03 22:46:31.764081 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:46:31.764061 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5\": container with ID starting with 769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5 not found: ID does not exist" containerID="769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5" Nov 03 22:46:31.764144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.764088 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5"} err="failed to get container status \"769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5\": rpc error: code = NotFound desc = could not find container \"769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5\": container with ID starting with 769c538e0faed10d01e04f85d91969770fb521ddd7001e6d01345090054893c5 not found: ID does not exist" Nov 03 22:46:31.764144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.764105 4898 scope.go:122] "RemoveContainer" containerID="faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2" Nov 03 22:46:31.764347 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:46:31.764327 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2\": container with ID starting with faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2 not found: ID does not exist" containerID="faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2" Nov 03 22:46:31.764400 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.764354 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2"} err="failed to get container status \"faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2\": rpc error: code = NotFound desc = could not find container \"faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2\": container with ID starting with faaff4627c7aa12a9a8383b8de2620d49055bc581585c37d976cba490391c7c2 not found: ID does not exist" Nov 03 22:46:31.911742 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.911711 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4"] Nov 03 22:46:31.962508 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:31.962477 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-027a6fbe-gw-openshift-default-ccd69c4b5-stjc4"] Nov 03 22:46:32.910932 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:46:32.910890 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" path="/var/lib/kubelet/pods/58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8/volumes" Nov 03 22:46:49.835438 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:46:49.835399 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:47:00.946935 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:00.946898 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r"] Nov 03 22:47:00.947505 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:00.947163 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" podUID="fff4c634-689d-4147-bcff-a9f548b6a65d" containerName="maas-api" containerID="cri-o://07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b" gracePeriod=30 Nov 03 22:47:01.425018 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.424301 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:47:01.529125 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.529086 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff4c634-689d-4147-bcff-a9f548b6a65d-kube-api-access-gwnwm\" (UniqueName: \"kubernetes.io/projected/fff4c634-689d-4147-bcff-a9f548b6a65d-kube-api-access-gwnwm\") pod \"fff4c634-689d-4147-bcff-a9f548b6a65d\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " Nov 03 22:47:01.529698 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.529665 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-metrics-certs\") pod \"fff4c634-689d-4147-bcff-a9f548b6a65d\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " Nov 03 22:47:01.529879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.529737 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\") pod \"fff4c634-689d-4147-bcff-a9f548b6a65d\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " Nov 03 22:47:01.529879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.529848 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff4c634-689d-4147-bcff-a9f548b6a65d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff4c634-689d-4147-bcff-a9f548b6a65d-openshift-service-ca\") pod \"fff4c634-689d-4147-bcff-a9f548b6a65d\" (UID: \"fff4c634-689d-4147-bcff-a9f548b6a65d\") " Nov 03 22:47:01.530967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.530935 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff4c634-689d-4147-bcff-a9f548b6a65d-openshift-service-ca" pod "fff4c634-689d-4147-bcff-a9f548b6a65d" (UID: "fff4c634-689d-4147-bcff-a9f548b6a65d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:01.536044 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.536007 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff4c634-689d-4147-bcff-a9f548b6a65d-kube-api-access-gwnwm" pod "fff4c634-689d-4147-bcff-a9f548b6a65d" (UID: "fff4c634-689d-4147-bcff-a9f548b6a65d"). InnerVolumeSpecName "kube-api-access-gwnwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:01.544410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.544373 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-metrics-certs" pod "fff4c634-689d-4147-bcff-a9f548b6a65d" (UID: "fff4c634-689d-4147-bcff-a9f548b6a65d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:01.545506 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.545476 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls" pod "fff4c634-689d-4147-bcff-a9f548b6a65d" (UID: "fff4c634-689d-4147-bcff-a9f548b6a65d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:01.632334 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.632239 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:01.632334 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.632275 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff4c634-689d-4147-bcff-a9f548b6a65d-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:01.632334 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.632289 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff4c634-689d-4147-bcff-a9f548b6a65d-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:01.632334 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:01.632303 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnwm\" (UniqueName: \"kubernetes.io/projected/fff4c634-689d-4147-bcff-a9f548b6a65d-kube-api-access-gwnwm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:02.035869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.035442 4898 generic.go:356] "Generic (PLEG): container finished" podID="fff4c634-689d-4147-bcff-a9f548b6a65d" containerID="07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b" exitCode=0 Nov 03 22:47:02.035869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.035699 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" event={"ID":"fff4c634-689d-4147-bcff-a9f548b6a65d","Type":"ContainerDied","Data":"07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b"} Nov 03 22:47:02.035869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.035745 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" event={"ID":"fff4c634-689d-4147-bcff-a9f548b6a65d","Type":"ContainerDied","Data":"2f3762f5b4c20be8d97c6e26660d0604ff11781870e2fc28e7eed6c7e0b8aa7d"} Nov 03 22:47:02.035869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.035772 4898 scope.go:122] "RemoveContainer" containerID="07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b" Nov 03 22:47:02.036510 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.035964 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r" Nov 03 22:47:02.071998 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.071964 4898 scope.go:122] "RemoveContainer" containerID="07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b" Nov 03 22:47:02.073503 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:47:02.073460 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b\": container with ID starting with 07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b not found: ID does not exist" containerID="07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b" Nov 03 22:47:02.073685 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.073655 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b"} err="failed to get container status \"07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b\": rpc error: code = NotFound desc = could not find container \"07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b\": container with ID starting with 07158583e66669f9ef7d2630b4e280a2567dec1d03e2bba5a91e455f7aa4d94b not found: ID does not exist" Nov 03 22:47:02.212965 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.212930 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r"] Nov 03 22:47:02.262952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.262910 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6905007-65798b8d5-bdf8r"] Nov 03 22:47:02.907844 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:02.907799 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff4c634-689d-4147-bcff-a9f548b6a65d" path="/var/lib/kubelet/pods/fff4c634-689d-4147-bcff-a9f548b6a65d/volumes" Nov 03 22:47:08.018089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018051 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p"] Nov 03 22:47:08.018683 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018669 4898 cpu_manager.go:414] "containerMap: removing container" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:47:08.018731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018683 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6653d7-4f7c-4f1b-852e-bec023b0968f" containerName="cleanup" Nov 03 22:47:08.018731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018718 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:47:08.018731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018723 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:47:08.018860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018732 4898 cpu_manager.go:414] "containerMap: removing container" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" Nov 03 22:47:08.018860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018737 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" Nov 03 22:47:08.018860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018758 4898 cpu_manager.go:414] "containerMap: removing container" podUID="fff4c634-689d-4147-bcff-a9f548b6a65d" containerName="maas-api" Nov 03 22:47:08.018860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018764 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff4c634-689d-4147-bcff-a9f548b6a65d" containerName="maas-api" Nov 03 22:47:08.018860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018774 4898 cpu_manager.go:414] "containerMap: removing container" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" Nov 03 22:47:08.018860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018778 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" Nov 03 22:47:08.019032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018921 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cdf855-ea27-4294-8f79-ab3a30589386" containerName="cleanup" Nov 03 22:47:08.019032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018932 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" Nov 03 22:47:08.019032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018938 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06d0b27-6f23-4ea3-9640-c3f0468e9a2a" containerName="cleanup" Nov 03 22:47:08.019032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018946 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff4c634-689d-4147-bcff-a9f548b6a65d" containerName="maas-api" Nov 03 22:47:08.019032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.018953 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c6a4e6-2b93-4f19-9e94-cf4fddd4d5d8" containerName="istio-proxy" Nov 03 22:47:08.023887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.023859 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.038996 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.038971 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-metrics-certs\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.039137 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.039123 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.039239 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.039220 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-openshift-service-ca\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.039332 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.039299 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2l7\" (UniqueName: \"kubernetes.io/projected/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-kube-api-access-lt2l7\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.079360 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.079321 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p"] Nov 03 22:47:08.140849 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.140814 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-metrics-certs\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.141013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.140993 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.141065 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.141049 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-openshift-service-ca\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.141107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.141084 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2l7\" (UniqueName: \"kubernetes.io/projected/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-kube-api-access-lt2l7\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.141859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.141833 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-openshift-service-ca\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.144145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.144123 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-metrics-certs\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.175544 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.175515 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e7a91ca4-serving-cert\"" Nov 03 22:47:08.175944 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:47:08.175913 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-serving-cert: secret "maas-api-e2e-derive-e7a91ca4-serving-cert" not found Nov 03 22:47:08.176329 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:47:08.176301 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls podName:d7042e00-7cf6-4fd0-a928-ad7dfa21cda3 nodeName:}" failed. No retries permitted until 2026-11-03 22:47:08.676278716 +0000 UTC m=+1471.889047748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls") pod "maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" (UID: "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3") : secret "maas-api-e2e-derive-e7a91ca4-serving-cert" not found Nov 03 22:47:08.203276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.203205 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2l7\" (UniqueName: \"kubernetes.io/projected/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-kube-api-access-lt2l7\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.755107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.755069 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.757609 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.757578 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls\") pod \"maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:08.938616 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:08.938580 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:09.343751 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:09.343718 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p"] Nov 03 22:47:09.344985 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:47:09.344951 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7042e00_7cf6_4fd0_a928_ad7dfa21cda3.slice/crio-7e74687a1837e18f33f2cbe7cdc6171c2341d275e6e17fad3bcf6e261f2d9b6d WatchSource:0}: Error finding container 7e74687a1837e18f33f2cbe7cdc6171c2341d275e6e17fad3bcf6e261f2d9b6d: Status 404 returned error can't find the container with id 7e74687a1837e18f33f2cbe7cdc6171c2341d275e6e17fad3bcf6e261f2d9b6d Nov 03 22:47:10.249978 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:10.249939 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" event={"ID":"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3","Type":"ContainerStarted","Data":"7e74687a1837e18f33f2cbe7cdc6171c2341d275e6e17fad3bcf6e261f2d9b6d"} Nov 03 22:47:11.260228 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:11.260188 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" event={"ID":"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3","Type":"ContainerStarted","Data":"59ef9924ef134407ec25de04e2f6514805c1dbae8ace696b6fabb7f29f153310"} Nov 03 22:47:11.260731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:11.260287 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:11.379613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:11.379549 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" podStartSLOduration=3.437462316 podStartE2EDuration="4.379534758s" totalImagesPullingTime="942.072442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:11.378046487 +0000 UTC m=+1474.590815563" watchObservedRunningTime="2026-11-03 22:47:11.379534758 +0000 UTC m=+1474.592303842" Nov 03 22:47:17.273333 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:17.273242 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:47:18.753807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.753763 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47"] Nov 03 22:47:18.759569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.759542 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.790836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.790759 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.791039 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.790862 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad6350-f4b1-4479-b30a-72c77d8737ac-openshift-service-ca\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.791039 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.790935 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-metrics-certs\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.791039 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.790972 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpcxr\" (UniqueName: \"kubernetes.io/projected/21ad6350-f4b1-4479-b30a-72c77d8737ac-kube-api-access-vpcxr\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.829223 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.829185 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47"] Nov 03 22:47:18.865450 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.865416 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d18a5b05-serving-cert\"" Nov 03 22:47:18.892153 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.892120 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.892337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.892163 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad6350-f4b1-4479-b30a-72c77d8737ac-openshift-service-ca\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.892337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.892284 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-metrics-certs\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.892337 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:47:18.892292 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-serving-cert: secret "maas-api-e2e-mt-d18a5b05-serving-cert" not found Nov 03 22:47:18.892505 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.892341 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpcxr\" (UniqueName: \"kubernetes.io/projected/21ad6350-f4b1-4479-b30a-72c77d8737ac-kube-api-access-vpcxr\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.892505 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:47:18.892378 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls podName:21ad6350-f4b1-4479-b30a-72c77d8737ac nodeName:}" failed. No retries permitted until 2026-11-03 22:47:19.392354914 +0000 UTC m=+1482.605123965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls") pod "maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" (UID: "21ad6350-f4b1-4479-b30a-72c77d8737ac") : secret "maas-api-e2e-mt-d18a5b05-serving-cert" not found Nov 03 22:47:18.892985 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.892956 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad6350-f4b1-4479-b30a-72c77d8737ac-openshift-service-ca\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.895874 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.895852 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-metrics-certs\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:18.954354 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:18.954320 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpcxr\" (UniqueName: \"kubernetes.io/projected/21ad6350-f4b1-4479-b30a-72c77d8737ac-kube-api-access-vpcxr\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:19.398329 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:19.398295 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:19.400735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:19.400716 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls\") pod \"maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:19.675406 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:19.675321 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:19.836194 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:47:19.836157 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:47:20.057456 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:20.057419 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47"] Nov 03 22:47:20.059491 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:47:20.058399 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ad6350_f4b1_4479_b30a_72c77d8737ac.slice/crio-af7e3a6a4041cacb877d02df87d89ff754d6dcf44ec0cf15984af2408f90054f WatchSource:0}: Error finding container af7e3a6a4041cacb877d02df87d89ff754d6dcf44ec0cf15984af2408f90054f: Status 404 returned error can't find the container with id af7e3a6a4041cacb877d02df87d89ff754d6dcf44ec0cf15984af2408f90054f Nov 03 22:47:20.328521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:20.328477 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" event={"ID":"21ad6350-f4b1-4479-b30a-72c77d8737ac","Type":"ContainerStarted","Data":"af7e3a6a4041cacb877d02df87d89ff754d6dcf44ec0cf15984af2408f90054f"} Nov 03 22:47:20.947640 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:20.947598 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj"] Nov 03 22:47:20.948183 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:20.947902 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" containerID="cri-o://3dccd7923879a872ef08612faa72c92e284149c08315115a5f770ec6aa9cd5e4" gracePeriod=30 Nov 03 22:47:21.341188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:21.341136 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" event={"ID":"21ad6350-f4b1-4479-b30a-72c77d8737ac","Type":"ContainerStarted","Data":"071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602"} Nov 03 22:47:21.341444 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:21.341419 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:21.462482 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:21.462423 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" podStartSLOduration=2.567160791 podStartE2EDuration="3.462407945s" totalImagesPullingTime="895.247154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:21.459074196 +0000 UTC m=+1484.671843267" watchObservedRunningTime="2026-11-03 22:47:21.462407945 +0000 UTC m=+1484.675176996" Nov 03 22:47:23.244335 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.244299 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt"] Nov 03 22:47:23.249073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.249049 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.293475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.293430 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm"] Nov 03 22:47:23.297852 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.297834 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.323733 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.323708 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5vrs\"" Nov 03 22:47:23.345009 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.344967 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt"] Nov 03 22:47:23.367011 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.366969 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.367465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.367445 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.367767 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.367751 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.368053 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.368031 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.368368 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.368351 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.368751 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.368732 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcg98\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-kube-api-access-qcg98\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.370511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370486 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.370646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370539 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.370646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370599 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69134f35-56c1-4e30-95e2-8989973e6a95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.370763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370656 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.370763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370684 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.370763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370720 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.370763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370753 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.371022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370862 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.371022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370889 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69134f35-56c1-4e30-95e2-8989973e6a95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.371022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370925 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.371022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.370953 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.371022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.371018 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvzg\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-kube-api-access-8vvzg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.389903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.389867 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm"] Nov 03 22:47:23.486474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486434 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.486474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486486 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69134f35-56c1-4e30-95e2-8989973e6a95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.486729 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486523 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.486729 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486550 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.486729 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486615 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvzg\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-kube-api-access-8vvzg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.486729 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486668 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.486957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486761 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.486957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486825 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.487073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486951 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.487073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.486986 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.487073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487052 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcg98\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-kube-api-access-qcg98\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.487073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487069 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.487282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487103 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.487282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487150 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69134f35-56c1-4e30-95e2-8989973e6a95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.487282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487199 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.487282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487228 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.487282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487251 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.488344 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.487279 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.488504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.488483 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.489270 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.489246 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.500172 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.489676 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.500326 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.490514 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69134f35-56c1-4e30-95e2-8989973e6a95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.500425 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.491276 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.500512 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.491499 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.500605 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.491708 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.500704 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.492310 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.500813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.492416 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.500947 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.496917 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.501035 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.500079 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.510690 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.510337 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.525912 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.520735 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.525912 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.523276 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69134f35-56c1-4e30-95e2-8989973e6a95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.665810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.655675 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.665810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.658457 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvzg\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-kube-api-access-8vvzg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.693382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.693310 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n5gnt\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.744812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.741594 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcg98\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-kube-api-access-qcg98\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k2jmm\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:23.865133 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.865073 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:23.912718 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:23.912682 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:24.361402 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:47:24.361346 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69134f35_56c1_4e30_95e2_8989973e6a95.slice/crio-f2e5fd93cbd3d94b6da6cae688e1806fe1f274ec9bba46a70eded7cdff55f4d7 WatchSource:0}: Error finding container f2e5fd93cbd3d94b6da6cae688e1806fe1f274ec9bba46a70eded7cdff55f4d7: Status 404 returned error can't find the container with id f2e5fd93cbd3d94b6da6cae688e1806fe1f274ec9bba46a70eded7cdff55f4d7 Nov 03 22:47:24.363045 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.363011 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt"] Nov 03 22:47:24.365486 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.365445 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.365587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.365533 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.365587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.365575 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.382320 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.382289 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" event={"ID":"69134f35-56c1-4e30-95e2-8989973e6a95","Type":"ContainerStarted","Data":"f2e5fd93cbd3d94b6da6cae688e1806fe1f274ec9bba46a70eded7cdff55f4d7"} Nov 03 22:47:24.415920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.415893 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm"] Nov 03 22:47:24.417477 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.417439 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.417576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.417520 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.417576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:24.417563 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:25.392145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.392110 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" event={"ID":"69134f35-56c1-4e30-95e2-8989973e6a95","Type":"ContainerStarted","Data":"1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86"} Nov 03 22:47:25.393809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.393769 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" event={"ID":"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0","Type":"ContainerStarted","Data":"522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403"} Nov 03 22:47:25.393923 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.393823 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" event={"ID":"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0","Type":"ContainerStarted","Data":"0ec086f1483f49e139185c370b4705ee3a7812a9494edbdd721d8e363b66166c"} Nov 03 22:47:25.579630 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.579517 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podStartSLOduration=2.579493817 podStartE2EDuration="2.579493817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:25.566244105 +0000 UTC m=+1488.779013158" watchObservedRunningTime="2026-11-03 22:47:25.579493817 +0000 UTC m=+1488.792262870" Nov 03 22:47:25.705919 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.699942 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podStartSLOduration=2.699919966 podStartE2EDuration="2.699919966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:25.698393256 +0000 UTC m=+1488.911162309" watchObservedRunningTime="2026-11-03 22:47:25.699919966 +0000 UTC m=+1488.912689022" Nov 03 22:47:25.877813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.872454 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:25.895829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.893932 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:25.895829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.893996 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:25.913281 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.913246 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:25.914763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.914732 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:25.914926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:25.914783 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:26.411037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:26.411004 4898 generic.go:356] "Generic (PLEG): container finished" podID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerID="3dccd7923879a872ef08612faa72c92e284149c08315115a5f770ec6aa9cd5e4" exitCode=0 Nov 03 22:47:26.411539 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:26.411081 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" event={"ID":"30088937-2aae-4df7-b461-bdb0889bf8a1","Type":"ContainerDied","Data":"3dccd7923879a872ef08612faa72c92e284149c08315115a5f770ec6aa9cd5e4"} Nov 03 22:47:26.868815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:26.868195 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:26.868815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:26.868256 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:26.913514 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:26.913476 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:26.913774 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:26.913745 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:27.310906 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.310700 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:47:27.390817 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.387014 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:47:27.418462 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418430 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-kube-api-access-492n6\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-kube-api-access-492n6\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418598 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-socket\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418658 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30088937-2aae-4df7-b461-bdb0889bf8a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30088937-2aae-4df7-b461-bdb0889bf8a1-istiod-ca-cert\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418710 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-credential-socket\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418749 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-certs\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418836 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-data\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418880 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-socket" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.418899 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-token\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-token\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.419006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419009 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-podinfo\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419040 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-envoy\") pod \"30088937-2aae-4df7-b461-bdb0889bf8a1\" (UID: \"30088937-2aae-4df7-b461-bdb0889bf8a1\") " Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419089 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-certs" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419224 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-credential-socket" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419424 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-data" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419596 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30088937-2aae-4df7-b461-bdb0889bf8a1-istiod-ca-cert" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419777 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" event={"ID":"30088937-2aae-4df7-b461-bdb0889bf8a1","Type":"ContainerDied","Data":"cb1cd3a1085b17f68064d4c5d808178c67bd5187014ee802bb488fd0257f5a1b"} Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419850 4898 scope.go:122] "RemoveContainer" containerID="3dccd7923879a872ef08612faa72c92e284149c08315115a5f770ec6aa9cd5e4" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419860 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.419982 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.420020 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30088937-2aae-4df7-b461-bdb0889bf8a1-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.420060 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.420076 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.421063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.420090 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.422342 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.421906 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-kube-api-access-492n6" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "kube-api-access-492n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:27.422342 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.422140 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-envoy" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:27.423860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.423841 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-podinfo" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:27.424886 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.423999 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-token" pod "30088937-2aae-4df7-b461-bdb0889bf8a1" (UID: "30088937-2aae-4df7-b461-bdb0889bf8a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:27.521857 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.521824 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492n6\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-kube-api-access-492n6\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.521857 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.521859 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.522048 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.521879 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.522048 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.521895 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30088937-2aae-4df7-b461-bdb0889bf8a1-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:27.877908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.877871 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:27.878167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.878137 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:27.908808 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.908755 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj"] Nov 03 22:47:27.914712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.914129 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:27.914712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.914186 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:27.916936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:27.916893 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-x5fmj"] Nov 03 22:47:28.877814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:28.867141 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:28.877814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:28.867206 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:28.911756 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:28.911724 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" path="/var/lib/kubelet/pods/30088937-2aae-4df7-b461-bdb0889bf8a1/volumes" Nov 03 22:47:28.913444 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:28.913422 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:28.913525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:28.913474 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:29.873811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:29.873141 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:29.873811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:29.873205 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:29.913349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:29.913314 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:29.913740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:29.913363 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:30.899810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:30.898914 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:30.899810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:30.898983 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:30.915984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:30.915947 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:30.916438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:30.916004 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:31.934813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:31.927608 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:31.934813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:31.927746 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:31.934813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:31.931146 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:31.934813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:31.931197 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:32.866805 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:32.866748 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n5gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:47:32.867130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:32.866856 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:47:32.913949 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:32.913915 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k2jmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:47:32.914113 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:32.913978 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:47:34.437635 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:34.437606 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:34.439087 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:34.439061 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:34.440432 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:34.440411 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:47:34.645477 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:34.645443 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:34.646452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:34.646423 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:34.647944 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:34.647918 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:47:37.249402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:37.249367 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:47:37.252572 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:37.252545 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:47:37.255159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:37.255136 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:47:37.257920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:47:37.257900 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:47:49.835947 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:47:49.835902 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:48:08.599630 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.599586 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw"] Nov 03 22:48:08.600580 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.600555 4898 cpu_manager.go:414] "containerMap: removing container" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" Nov 03 22:48:08.600690 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.600581 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" Nov 03 22:48:08.600771 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.600756 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="30088937-2aae-4df7-b461-bdb0889bf8a1" containerName="istio-proxy" Nov 03 22:48:08.604526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.604504 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.650539 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.650500 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45"] Nov 03 22:48:08.654902 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.654881 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.745428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.745390 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45"] Nov 03 22:48:08.772227 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772184 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.772418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772299 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.772418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772385 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.772555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772449 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.772555 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772542 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2twn\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-kube-api-access-h2twn\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.772673 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772588 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.772673 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772639 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.772781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772695 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.772781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772733 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.772916 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772764 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.772972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772905 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxk8\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-kube-api-access-tdxk8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.772972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772939 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.773118 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.772989 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.773118 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.773016 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.773118 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.773086 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.773285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.773125 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.773285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.773164 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.773285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.773225 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.810991 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.810932 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw"] Nov 03 22:48:08.874316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874225 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.874316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874259 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.874316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874289 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2twn\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-kube-api-access-h2twn\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.874602 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874348 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.874602 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874374 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.874602 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874398 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.874602 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874423 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.874825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874728 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.874890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874817 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.874942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874881 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.874942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874889 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxk8\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-kube-api-access-tdxk8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.874942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874917 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874944 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874981 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.875100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.874980 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875034 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.875100 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875073 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.875349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875113 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.875349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875173 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875259 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875284 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875294 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.875578 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875563 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.875626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875574 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875624 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.875804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.875768 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.876197 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.876172 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.876299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.876233 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.877127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.877101 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.878132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.878104 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.878440 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.878423 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.878583 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.878567 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.936812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.936700 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxk8\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-kube-api-access-tdxk8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.936812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.936719 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.936812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.936760 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ddv45\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:08.937107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.936892 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2twn\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-kube-api-access-h2twn\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fkwnw\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:08.969761 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:08.969729 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:09.218842 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.218754 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:09.408936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.405332 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45"] Nov 03 22:48:09.409396 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:48:09.409366 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa713a28_16bb_48bc_8ba6_e4cbfdb2bd27.slice/crio-35cacbcce9d8374f11b782663dee7a4ada2f8a7085f8b5eefb4877b755f82581 WatchSource:0}: Error finding container 35cacbcce9d8374f11b782663dee7a4ada2f8a7085f8b5eefb4877b755f82581: Status 404 returned error can't find the container with id 35cacbcce9d8374f11b782663dee7a4ada2f8a7085f8b5eefb4877b755f82581 Nov 03 22:48:09.412521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.412488 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.412608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.412554 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.412608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.412596 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.620321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.620282 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw"] Nov 03 22:48:09.624600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.624559 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.624691 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.624647 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.624730 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.624692 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.885902 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.885844 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" event={"ID":"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117","Type":"ContainerStarted","Data":"66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849"} Nov 03 22:48:09.886219 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.886175 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" event={"ID":"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117","Type":"ContainerStarted","Data":"0eab286540bbb3f220cd350c68b306ee8f34592f46356f104110d455f9a56c45"} Nov 03 22:48:09.888222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.888187 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" event={"ID":"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27","Type":"ContainerStarted","Data":"296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02"} Nov 03 22:48:09.888337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.888238 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" event={"ID":"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27","Type":"ContainerStarted","Data":"35cacbcce9d8374f11b782663dee7a4ada2f8a7085f8b5eefb4877b755f82581"} Nov 03 22:48:09.970447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:09.970398 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:10.009282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:10.009223 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podStartSLOduration=2.009209643 podStartE2EDuration="2.009209643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:10.007125679 +0000 UTC m=+1533.219894746" watchObservedRunningTime="2026-11-03 22:48:10.009209643 +0000 UTC m=+1533.221978693" Nov 03 22:48:10.132779 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:10.132421 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podStartSLOduration=2.132402793 podStartE2EDuration="2.132402793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:10.128132721 +0000 UTC m=+1533.340901773" watchObservedRunningTime="2026-11-03 22:48:10.132402793 +0000 UTC m=+1533.345171845" Nov 03 22:48:10.219461 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:10.219368 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:10.220648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:10.220622 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:10.220775 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:10.220671 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:10.991289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:10.985525 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:48:10.991289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:10.985593 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 22:48:11.219538 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:11.219487 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:11.219538 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:11.219543 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:11.984831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:11.972160 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:48:11.984831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:11.972219 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 22:48:12.219776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:12.219741 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:12.220187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:12.219799 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:12.971625 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:12.971174 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:48:12.971625 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:12.971235 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 22:48:13.219665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:13.219626 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:13.219910 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:13.219684 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:13.970403 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:13.970368 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:48:13.970703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:13.970673 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 22:48:14.219670 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:14.219625 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:14.219910 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:14.219685 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:14.978161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:14.975678 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:48:14.978161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:14.976000 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 22:48:15.219387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:15.219348 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:15.219548 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:15.219408 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:15.970985 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:15.970953 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:48:15.971391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:15.971010 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 22:48:16.219808 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:16.219756 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:16.220001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:16.219840 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:16.971300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:16.971260 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:48:16.971838 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:16.971332 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 22:48:17.219457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:17.219421 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:48:17.219618 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:17.219476 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 22:48:18.493432 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:18.493393 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:18.496551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:18.494975 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:18.496882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:18.496853 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:48:18.635615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:18.633990 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:18.635615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:18.634383 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:18.636014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:48:18.635993 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:48:19.839005 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:48:19.838963 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:48:49.836105 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:48:49.836064 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:49:14.187277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:14.187239 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p"] Nov 03 22:49:14.187721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:14.187509 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" podUID="d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" containerName="maas-api" containerID="cri-o://59ef9924ef134407ec25de04e2f6514805c1dbae8ace696b6fabb7f29f153310" gracePeriod=30 Nov 03 22:49:14.756528 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:14.756445 4898 generic.go:356] "Generic (PLEG): container finished" podID="d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" containerID="59ef9924ef134407ec25de04e2f6514805c1dbae8ace696b6fabb7f29f153310" exitCode=0 Nov 03 22:49:14.756972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:14.756901 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" event={"ID":"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3","Type":"ContainerDied","Data":"59ef9924ef134407ec25de04e2f6514805c1dbae8ace696b6fabb7f29f153310"} Nov 03 22:49:14.970183 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:14.959199 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:49:15.052276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.052180 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls\") pod \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " Nov 03 22:49:15.052450 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.052390 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-metrics-certs\") pod \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " Nov 03 22:49:15.052516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.052451 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-openshift-service-ca\") pod \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " Nov 03 22:49:15.052591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.052571 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-kube-api-access-lt2l7\" (UniqueName: \"kubernetes.io/projected/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-kube-api-access-lt2l7\") pod \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\" (UID: \"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3\") " Nov 03 22:49:15.052923 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.052882 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-openshift-service-ca" pod "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" (UID: "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:15.053358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.053334 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:15.054739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.054709 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls" pod "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" (UID: "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:15.054846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.054816 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-kube-api-access-lt2l7" pod "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" (UID: "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3"). InnerVolumeSpecName "kube-api-access-lt2l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:15.055090 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.055076 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-metrics-certs" pod "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" (UID: "d7042e00-7cf6-4fd0-a928-ad7dfa21cda3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:15.154319 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.154285 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:15.154319 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.154315 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt2l7\" (UniqueName: \"kubernetes.io/projected/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-kube-api-access-lt2l7\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:15.154518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.154329 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:15.766696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.766647 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" event={"ID":"d7042e00-7cf6-4fd0-a928-ad7dfa21cda3","Type":"ContainerDied","Data":"7e74687a1837e18f33f2cbe7cdc6171c2341d275e6e17fad3bcf6e261f2d9b6d"} Nov 03 22:49:15.767193 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.766667 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p" Nov 03 22:49:15.767193 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.766771 4898 scope.go:122] "RemoveContainer" containerID="59ef9924ef134407ec25de04e2f6514805c1dbae8ace696b6fabb7f29f153310" Nov 03 22:49:15.935437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.935397 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p"] Nov 03 22:49:15.985332 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:15.985293 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7a91ca4-5cb95d6799-rqv7p"] Nov 03 22:49:16.907704 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:16.907671 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" path="/var/lib/kubelet/pods/d7042e00-7cf6-4fd0-a928-ad7dfa21cda3/volumes" Nov 03 22:49:19.838416 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:49:19.838373 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:49:22.468096 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.468060 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh"] Nov 03 22:49:22.468676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.468300 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" containerID="cri-o://b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6" gracePeriod=30 Nov 03 22:49:22.519363 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.519333 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k"] Nov 03 22:49:22.519572 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.519545 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" containerID="cri-o://9ee0438ac8f030465e7dee929457e9742a1082af3e88a18f752eefe258668151" gracePeriod=30 Nov 03 22:49:22.525281 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.525255 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst"] Nov 03 22:49:22.525715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.525632 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" containerID="cri-o://e739971d611b2f74443118aa2c93be71cb1b5ce2dc6fbe73e050c72cebb49f95" gracePeriod=30 Nov 03 22:49:22.570314 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.570271 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz"] Nov 03 22:49:22.570552 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:22.570523 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" containerID="cri-o://a41330f97148284f490910856e8ceb91b0f73740c8d2378e60b3fca55e232c07" gracePeriod=30 Nov 03 22:49:24.888246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:24.888209 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:49:24.888613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:24.888269 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:49:25.214349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:25.214261 4898 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:49:25.214349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:25.214321 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:49:27.827515 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.827488 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:49:27.901301 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.901263 4898 generic.go:356] "Generic (PLEG): container finished" podID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerID="a41330f97148284f490910856e8ceb91b0f73740c8d2378e60b3fca55e232c07" exitCode=0 Nov 03 22:49:27.901520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.901389 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" event={"ID":"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0","Type":"ContainerDied","Data":"a41330f97148284f490910856e8ceb91b0f73740c8d2378e60b3fca55e232c07"} Nov 03 22:49:27.903651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.903620 4898 generic.go:356] "Generic (PLEG): container finished" podID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerID="e739971d611b2f74443118aa2c93be71cb1b5ce2dc6fbe73e050c72cebb49f95" exitCode=0 Nov 03 22:49:27.903783 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.903745 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" event={"ID":"1c8ee53f-509e-429a-b430-23f828e0cfb0","Type":"ContainerDied","Data":"e739971d611b2f74443118aa2c93be71cb1b5ce2dc6fbe73e050c72cebb49f95"} Nov 03 22:49:27.905348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.905326 4898 generic.go:356] "Generic (PLEG): container finished" podID="48419746-5dd4-4d86-8382-84ec8283fe40" containerID="b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6" exitCode=0 Nov 03 22:49:27.905443 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.905384 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" Nov 03 22:49:27.905443 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.905402 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" event={"ID":"48419746-5dd4-4d86-8382-84ec8283fe40","Type":"ContainerDied","Data":"b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6"} Nov 03 22:49:27.905567 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.905446 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh" event={"ID":"48419746-5dd4-4d86-8382-84ec8283fe40","Type":"ContainerDied","Data":"fee1480320695e44eacac23c33a926e6d1fe8b2bf874ab4cc3e839d32eba0ca0"} Nov 03 22:49:27.905567 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.905476 4898 scope.go:122] "RemoveContainer" containerID="b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6" Nov 03 22:49:27.907294 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.907275 4898 generic.go:356] "Generic (PLEG): container finished" podID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerID="9ee0438ac8f030465e7dee929457e9742a1082af3e88a18f752eefe258668151" exitCode=0 Nov 03 22:49:27.907371 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.907325 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" event={"ID":"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1","Type":"ContainerDied","Data":"9ee0438ac8f030465e7dee929457e9742a1082af3e88a18f752eefe258668151"} Nov 03 22:49:27.925462 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.925441 4898 scope.go:122] "RemoveContainer" containerID="b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6" Nov 03 22:49:27.925952 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:49:27.925927 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6\": container with ID starting with b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6 not found: ID does not exist" containerID="b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6" Nov 03 22:49:27.926065 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.925965 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6"} err="failed to get container status \"b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6\": rpc error: code = NotFound desc = could not find container \"b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6\": container with ID starting with b7bc163158184be7476a7f3299a9e078b18a8d9c7082d9919e3ab300d4a1c3a6 not found: ID does not exist" Nov 03 22:49:27.940156 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.940132 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:49:27.946942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.946921 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:49:27.953345 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953320 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-data\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.953488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953475 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-envoy\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.953609 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953584 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-data" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:27.953668 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953641 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-socket\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.953724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953681 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48419746-5dd4-4d86-8382-84ec8283fe40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48419746-5dd4-4d86-8382-84ec8283fe40-istiod-ca-cert\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.953724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953717 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48419746-5dd4-4d86-8382-84ec8283fe40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48419746-5dd4-4d86-8382-84ec8283fe40-istio-podinfo\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.953878 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953763 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-certs\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.953878 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953831 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-kube-api-access-n5zg5\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-kube-api-access-n5zg5\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.954041 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953917 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-istio-token\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-istio-token\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.954041 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.953974 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-credential-socket\") pod \"48419746-5dd4-4d86-8382-84ec8283fe40\" (UID: \"48419746-5dd4-4d86-8382-84ec8283fe40\") " Nov 03 22:49:27.954453 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.954406 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-certs" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:27.954668 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.954644 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-credential-socket" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:27.954858 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.954818 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48419746-5dd4-4d86-8382-84ec8283fe40-istiod-ca-cert" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:27.954961 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.954864 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:27.954961 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.954896 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:27.954961 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.954910 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:27.955378 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.955353 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-socket" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:27.958958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.958921 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48419746-5dd4-4d86-8382-84ec8283fe40-istio-podinfo" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:27.958958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.958935 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-envoy" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:27.959128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.958953 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-kube-api-access-n5zg5" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "kube-api-access-n5zg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:27.959128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.958985 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-istio-token" pod "48419746-5dd4-4d86-8382-84ec8283fe40" (UID: "48419746-5dd4-4d86-8382-84ec8283fe40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:27.980671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:27.980651 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:49:28.056520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056433 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8ee53f-509e-429a-b430-23f828e0cfb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ee53f-509e-429a-b430-23f828e0cfb0-istiod-ca-cert\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.056520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056485 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-credential-socket\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.056520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056517 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-certs\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.056821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056544 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-token\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-token\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.056821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056583 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-credential-socket\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.056821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056619 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-kube-api-access-bcqhw\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-kube-api-access-bcqhw\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.056821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056649 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-kube-api-access-4lhmd\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-kube-api-access-4lhmd\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.056821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056728 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istiod-ca-cert\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.056821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056779 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-credential-socket\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056828 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-certs\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056780 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-credential-socket" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056878 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-podinfo\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056885 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-certs" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056891 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-credential-socket" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056906 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istiod-ca-cert\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.056978 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-socket\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057013 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-socket\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057124 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-certs\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057157 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-certs" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.057171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057169 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-data\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057205 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istiod-ca-cert" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057310 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-podinfo\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057349 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-envoy\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057367 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-credential-socket" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057378 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-envoy\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057415 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-kube-api-access-lcmjb\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-kube-api-access-lcmjb\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057453 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-podinfo\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057557 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-socket\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057577 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-certs" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057596 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-token\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-token\") pod \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\" (UID: \"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0\") " Nov 03 22:49:28.057715 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057662 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-data\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.058355 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057747 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-token\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.058355 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057803 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-envoy\") pod \"1c8ee53f-509e-429a-b430-23f828e0cfb0\" (UID: \"1c8ee53f-509e-429a-b430-23f828e0cfb0\") " Nov 03 22:49:28.058355 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057859 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-data\") pod \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\" (UID: \"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1\") " Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058635 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058661 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058679 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058692 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058707 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48419746-5dd4-4d86-8382-84ec8283fe40-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058720 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48419746-5dd4-4d86-8382-84ec8283fe40-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058734 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48419746-5dd4-4d86-8382-84ec8283fe40-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058748 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058762 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zg5\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-kube-api-access-n5zg5\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058777 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058831 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058846 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48419746-5dd4-4d86-8382-84ec8283fe40-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.058888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058859 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.061928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057802 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-socket" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.061928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.057819 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istiod-ca-cert" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:28.061928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058000 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-socket" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.061928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058022 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8ee53f-509e-429a-b430-23f828e0cfb0-istiod-ca-cert" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:28.062315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058756 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-data" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.062315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.058960 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-socket" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.062315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.059072 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-data" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.062315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.061326 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-data" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.065839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.065226 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-envoy" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.065839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.065707 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-token" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.065839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.065742 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-kube-api-access-bcqhw" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "kube-api-access-bcqhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.066073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.065980 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-kube-api-access-lcmjb" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "kube-api-access-lcmjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.066280 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.066242 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-envoy" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.066365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.066271 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-token" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.066365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.066354 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-podinfo" pod "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" (UID: "0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:28.067387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.067361 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-podinfo" pod "1c8ee53f-509e-429a-b430-23f828e0cfb0" (UID: "1c8ee53f-509e-429a-b430-23f828e0cfb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:28.067387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.067372 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-envoy" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.067536 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.067364 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-kube-api-access-4lhmd" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "kube-api-access-4lhmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.067536 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.067400 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-token" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.067754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.067732 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-podinfo" pod "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" (UID: "985df9f5-baaa-48c6-b7a2-438ddcb8b3e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:28.160220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160179 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160220 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160236 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160252 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160271 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160284 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160299 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160314 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ee53f-509e-429a-b430-23f828e0cfb0-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160327 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160343 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcqhw\" (UniqueName: \"kubernetes.io/projected/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-kube-api-access-bcqhw\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160359 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lhmd\" (UniqueName: \"kubernetes.io/projected/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-kube-api-access-4lhmd\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160374 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160388 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160403 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160418 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ee53f-509e-429a-b430-23f828e0cfb0-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160432 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160448 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ee53f-509e-429a-b430-23f828e0cfb0-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.160465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160463 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.161299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160478 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.161299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.160495 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcmjb\" (UniqueName: \"kubernetes.io/projected/1c8ee53f-509e-429a-b430-23f828e0cfb0-kube-api-access-lcmjb\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.548071 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.548029 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh"] Nov 03 22:49:28.552887 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.552859 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-hjqrh"] Nov 03 22:49:28.907118 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.907089 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" path="/var/lib/kubelet/pods/48419746-5dd4-4d86-8382-84ec8283fe40/volumes" Nov 03 22:49:28.926993 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.926960 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" Nov 03 22:49:28.927152 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.926969 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz" event={"ID":"0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0","Type":"ContainerDied","Data":"53b2523ab7870883ae5a4f4417197907d4e47c8f34969376762e9827a8d16cb1"} Nov 03 22:49:28.927152 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.927126 4898 scope.go:122] "RemoveContainer" containerID="a41330f97148284f490910856e8ceb91b0f73740c8d2378e60b3fca55e232c07" Nov 03 22:49:28.929990 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.929968 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" Nov 03 22:49:28.930374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.930000 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst" event={"ID":"1c8ee53f-509e-429a-b430-23f828e0cfb0","Type":"ContainerDied","Data":"d8c35df59f715c7342c26110b8bbd896a97517e0d77666521c64058730692416"} Nov 03 22:49:28.933261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.933232 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" event={"ID":"985df9f5-baaa-48c6-b7a2-438ddcb8b3e1","Type":"ContainerDied","Data":"cc8bc4338c7bf5f8c0ca414fe9d9840c2ff97888c2c4957c7a38aa5c697d53fa"} Nov 03 22:49:28.933261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.933248 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k" Nov 03 22:49:28.946419 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.946396 4898 scope.go:122] "RemoveContainer" containerID="e739971d611b2f74443118aa2c93be71cb1b5ce2dc6fbe73e050c72cebb49f95" Nov 03 22:49:28.958662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:28.958641 4898 scope.go:122] "RemoveContainer" containerID="9ee0438ac8f030465e7dee929457e9742a1082af3e88a18f752eefe258668151" Nov 03 22:49:29.142985 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:29.142921 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz"] Nov 03 22:49:29.282019 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:29.281853 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-lk9lz"] Nov 03 22:49:29.303370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:29.302868 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k"] Nov 03 22:49:29.307839 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:29.307778 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-krp2k"] Nov 03 22:49:29.387776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:29.387724 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst"] Nov 03 22:49:29.393680 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:29.393641 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-c6fst"] Nov 03 22:49:30.908725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:30.908693 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" path="/var/lib/kubelet/pods/0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0/volumes" Nov 03 22:49:30.909279 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:30.909263 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" path="/var/lib/kubelet/pods/1c8ee53f-509e-429a-b430-23f828e0cfb0/volumes" Nov 03 22:49:30.909870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:30.909852 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" path="/var/lib/kubelet/pods/985df9f5-baaa-48c6-b7a2-438ddcb8b3e1/volumes" Nov 03 22:49:49.088812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:49.087912 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:49:49.539872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:49.538891 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ddv45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:49:49.539872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:49.538977 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:49:49.707813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:49.697917 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fkwnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:49:49.707813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:49:49.698006 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:49:49.981533 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:49:49.979636 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:50:07.365135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.365102 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg"] Nov 03 22:50:07.365663 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.365344 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" containerID="cri-o://35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c" gracePeriod=30 Nov 03 22:50:07.424848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.419532 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk"] Nov 03 22:50:07.424848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.419876 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" containerID="cri-o://ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d" gracePeriod=30 Nov 03 22:50:07.424848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.424568 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd"] Nov 03 22:50:07.424848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.424758 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" containerID="cri-o://d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0" gracePeriod=30 Nov 03 22:50:07.468583 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.468516 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5"] Nov 03 22:50:07.468862 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:07.468830 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" containerID="cri-o://9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052" gracePeriod=30 Nov 03 22:50:09.039504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:09.039469 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:09.039937 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:09.039521 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:09.164429 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:09.164391 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:09.164591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:09.164457 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:11.336861 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:11.336826 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:11.337289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:11.336900 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:12.746369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.746327 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:50:12.815021 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.814980 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b96c70-3f89-4389-8e41-8f79ff7e3493-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b96c70-3f89-4389-8e41-8f79ff7e3493-istiod-ca-cert\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815047 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-certs\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815095 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-data\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815134 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-socket\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815183 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-envoy\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815315 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-credential-socket\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815359 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-kube-api-access-6kzpv\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-kube-api-access-6kzpv\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815404 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-token\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-token\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.815568 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.815522 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-podinfo\") pod \"62b96c70-3f89-4389-8e41-8f79ff7e3493\" (UID: \"62b96c70-3f89-4389-8e41-8f79ff7e3493\") " Nov 03 22:50:12.818010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.817973 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-certs" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.818909 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.818268 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-data" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.818909 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.818333 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-credential-socket" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.818909 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.818739 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b96c70-3f89-4389-8e41-8f79ff7e3493-istiod-ca-cert" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:12.818909 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.818817 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-socket" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.821702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.821663 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-token" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:12.822885 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.822857 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-podinfo" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:12.823316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.823283 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-envoy" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.824331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.824290 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-kube-api-access-6kzpv" pod "62b96c70-3f89-4389-8e41-8f79ff7e3493" (UID: "62b96c70-3f89-4389-8e41-8f79ff7e3493"). InnerVolumeSpecName "kube-api-access-6kzpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:12.858460 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.858431 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:50:12.912709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.912684 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.916862 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-kube-api-access-gj5kg\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-kube-api-access-gj5kg\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.916938 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/233153b9-fe39-4082-87c1-169894343e6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233153b9-fe39-4082-87c1-169894343e6b-istio-podinfo\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.916979 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-credential-socket\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917062 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-envoy\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917121 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-socket\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917149 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-data\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917186 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bba430-98f2-4ab6-a07f-a942ba4ad331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bba430-98f2-4ab6-a07f-a942ba4ad331-istiod-ca-cert\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917240 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-kube-api-access-sq5g2\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-kube-api-access-sq5g2\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917316 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-istio-token\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-istio-token\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917385 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-data\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917418 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-socket\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917462 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-certs\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917495 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233153b9-fe39-4082-87c1-169894343e6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233153b9-fe39-4082-87c1-169894343e6b-istiod-ca-cert\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917537 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-certs\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917580 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-credential-socket\") pod \"233153b9-fe39-4082-87c1-169894343e6b\" (UID: \"233153b9-fe39-4082-87c1-169894343e6b\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917642 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-token\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-token\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917672 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-envoy\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.917765 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-podinfo\") pod \"30bba430-98f2-4ab6-a07f-a942ba4ad331\" (UID: \"30bba430-98f2-4ab6-a07f-a942ba4ad331\") " Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918472 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918497 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b96c70-3f89-4389-8e41-8f79ff7e3493-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918513 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918529 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918546 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918561 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918576 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b96c70-3f89-4389-8e41-8f79ff7e3493-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918591 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzpv\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-kube-api-access-6kzpv\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.918606 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b96c70-3f89-4389-8e41-8f79ff7e3493-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.920034 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-certs" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.920335 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233153b9-fe39-4082-87c1-169894343e6b-istiod-ca-cert" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:12.920765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.920622 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-data" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.922439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.920905 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-certs" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.922439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.921221 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-data" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.922439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.921399 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-socket" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.922439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.921569 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-credential-socket" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.928812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.926229 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-podinfo" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:12.928812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.928521 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bba430-98f2-4ab6-a07f-a942ba4ad331-istiod-ca-cert" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:12.930211 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.930104 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-token" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:12.930611 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.930317 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-socket" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.930611 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.930506 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-credential-socket" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.930918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.930855 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-envoy" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.932311 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.932213 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:50:12.932311 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.932218 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-envoy" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:12.932311 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.932232 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-kube-api-access-gj5kg" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "kube-api-access-gj5kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:12.932881 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.932855 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-kube-api-access-sq5g2" pod "30bba430-98f2-4ab6-a07f-a942ba4ad331" (UID: "30bba430-98f2-4ab6-a07f-a942ba4ad331"). InnerVolumeSpecName "kube-api-access-sq5g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:12.933496 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.933422 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/233153b9-fe39-4082-87c1-169894343e6b-istio-podinfo" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:12.934149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:12.934128 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-istio-token" pod "233153b9-fe39-4082-87c1-169894343e6b" (UID: "233153b9-fe39-4082-87c1-169894343e6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:13.014894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.014779 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx"] Nov 03 22:50:13.015487 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015471 4898 cpu_manager.go:414] "containerMap: removing container" podUID="d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" containerName="maas-api" Nov 03 22:50:13.015546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015489 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" containerName="maas-api" Nov 03 22:50:13.015546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015508 4898 cpu_manager.go:414] "containerMap: removing container" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" Nov 03 22:50:13.015546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015514 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" Nov 03 22:50:13.015546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015522 4898 cpu_manager.go:414] "containerMap: removing container" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" Nov 03 22:50:13.015546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015528 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" Nov 03 22:50:13.015546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015546 4898 cpu_manager.go:414] "containerMap: removing container" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015551 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015558 4898 cpu_manager.go:414] "containerMap: removing container" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015563 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015572 4898 cpu_manager.go:414] "containerMap: removing container" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015577 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015583 4898 cpu_manager.go:414] "containerMap: removing container" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015589 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015619 4898 cpu_manager.go:414] "containerMap: removing container" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015625 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015635 4898 cpu_manager.go:414] "containerMap: removing container" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015641 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015737 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8ee53f-509e-429a-b430-23f828e0cfb0" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015744 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerName="istio-proxy" Nov 03 22:50:13.015747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015751 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerName="istio-proxy" Nov 03 22:50:13.016242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015758 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="48419746-5dd4-4d86-8382-84ec8283fe40" containerName="istio-proxy" Nov 03 22:50:13.016242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015766 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="985df9f5-baaa-48c6-b7a2-438ddcb8b3e1" containerName="istio-proxy" Nov 03 22:50:13.016242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015773 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b69acfc-5ad4-4a29-8d1a-f3e555f04cb0" containerName="istio-proxy" Nov 03 22:50:13.016242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015779 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7042e00-7cf6-4fd0-a928-ad7dfa21cda3" containerName="maas-api" Nov 03 22:50:13.016242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015799 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerName="istio-proxy" Nov 03 22:50:13.016242 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.015805 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="233153b9-fe39-4082-87c1-169894343e6b" containerName="istio-proxy" Nov 03 22:50:13.019635 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.019605 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istiod-ca-cert\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.019774 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.019755 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.019908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.019772 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-data\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.019908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.019825 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-certs\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.019908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.019871 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-socket\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.020085 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.019928 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-credential-socket\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.020085 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.019981 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-kube-api-access-cj5t2\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-kube-api-access-cj5t2\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.020085 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020047 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-data" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:13.020085 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020069 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istiod-ca-cert" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:13.020292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020106 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-token\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-token\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.020292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020177 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-podinfo\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.020292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020211 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-envoy\") pod \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\" (UID: \"d67caae8-c0f7-432a-bc11-7e72cb5750c4\") " Nov 03 22:50:13.020446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020352 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-socket" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020535 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-certs" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020761 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-credential-socket" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020894 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233153b9-fe39-4082-87c1-169894343e6b-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020918 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020936 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020952 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020966 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020980 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.020995 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5kg\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-kube-api-access-gj5kg\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021009 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021023 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233153b9-fe39-4082-87c1-169894343e6b-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021036 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021052 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021068 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021086 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021099 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bba430-98f2-4ab6-a07f-a942ba4ad331-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021112 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5g2\" (UniqueName: \"kubernetes.io/projected/30bba430-98f2-4ab6-a07f-a942ba4ad331-kube-api-access-sq5g2\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.021117 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021127 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.022014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021140 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.022014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021153 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.022014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021166 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233153b9-fe39-4082-87c1-169894343e6b-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.022014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021179 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.022014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021192 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bba430-98f2-4ab6-a07f-a942ba4ad331-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.022014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.021207 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233153b9-fe39-4082-87c1-169894343e6b-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.024026 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.024000 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-podinfo" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:13.024617 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.024591 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-kube-api-access-cj5t2" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "kube-api-access-cj5t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:13.024700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.024640 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-envoy" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:13.024935 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.024909 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-token" pod "d67caae8-c0f7-432a-bc11-7e72cb5750c4" (UID: "d67caae8-c0f7-432a-bc11-7e72cb5750c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:13.074129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.074097 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-fb4f0a-serving-cert\"" Nov 03 22:50:13.111317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.111278 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx"] Nov 03 22:50:13.122644 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.122604 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.122986 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.122964 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.123172 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.123155 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmpl\" (UniqueName: \"kubernetes.io/projected/c960bcf8-805c-4f86-a398-b476ad90ea44-kube-api-access-stmpl\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.123375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.123360 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c960bcf8-805c-4f86-a398-b476ad90ea44-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.123547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.123532 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.123768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.123666 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.123768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.123701 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67caae8-c0f7-432a-bc11-7e72cb5750c4-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.123768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.123718 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj5t2\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-kube-api-access-cj5t2\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.123768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.123745 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67caae8-c0f7-432a-bc11-7e72cb5750c4-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:13.225256 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.225216 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.225444 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.225304 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmpl\" (UniqueName: \"kubernetes.io/projected/c960bcf8-805c-4f86-a398-b476ad90ea44-kube-api-access-stmpl\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.225444 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:13.225388 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-serving-cert: secret "maas-api-e2e-shared-a-w2-fb4f0a-serving-cert" not found Nov 03 22:50:13.225444 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.225420 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c960bcf8-805c-4f86-a398-b476ad90ea44-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.225553 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:13.225473 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls podName:c960bcf8-805c-4f86-a398-b476ad90ea44 nodeName:}" failed. No retries permitted until 2026-11-03 22:50:13.725452659 +0000 UTC m=+1656.938221702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls") pod "maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" (UID: "c960bcf8-805c-4f86-a398-b476ad90ea44") : secret "maas-api-e2e-shared-a-w2-fb4f0a-serving-cert" not found Nov 03 22:50:13.225553 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.225498 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.226143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.226118 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c960bcf8-805c-4f86-a398-b476ad90ea44-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.228337 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.228312 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.301651 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.301566 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmpl\" (UniqueName: \"kubernetes.io/projected/c960bcf8-805c-4f86-a398-b476ad90ea44-kube-api-access-stmpl\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.387146 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.387112 4898 generic.go:356] "Generic (PLEG): container finished" podID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" containerID="9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052" exitCode=0 Nov 03 22:50:13.387352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.387168 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" Nov 03 22:50:13.387352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.387200 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" event={"ID":"d67caae8-c0f7-432a-bc11-7e72cb5750c4","Type":"ContainerDied","Data":"9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052"} Nov 03 22:50:13.387352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.387262 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5" event={"ID":"d67caae8-c0f7-432a-bc11-7e72cb5750c4","Type":"ContainerDied","Data":"867997befd2bf81fea5e9dbb410d61dfbe4f3c7158d808cee6a23d42b0a9b6e4"} Nov 03 22:50:13.387352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.387291 4898 scope.go:122] "RemoveContainer" containerID="9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052" Nov 03 22:50:13.388678 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.388658 4898 generic.go:356] "Generic (PLEG): container finished" podID="62b96c70-3f89-4389-8e41-8f79ff7e3493" containerID="35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c" exitCode=0 Nov 03 22:50:13.388782 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.388686 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" event={"ID":"62b96c70-3f89-4389-8e41-8f79ff7e3493","Type":"ContainerDied","Data":"35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c"} Nov 03 22:50:13.388782 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.388710 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" Nov 03 22:50:13.388782 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.388726 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg" event={"ID":"62b96c70-3f89-4389-8e41-8f79ff7e3493","Type":"ContainerDied","Data":"6b49f9ee173c1ec3be07cdb5a771f2c86a10ca99e9817642be75dbda6f01fd20"} Nov 03 22:50:13.390334 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.390302 4898 generic.go:356] "Generic (PLEG): container finished" podID="30bba430-98f2-4ab6-a07f-a942ba4ad331" containerID="ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d" exitCode=0 Nov 03 22:50:13.390437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.390352 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" Nov 03 22:50:13.390437 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.390393 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" event={"ID":"30bba430-98f2-4ab6-a07f-a942ba4ad331","Type":"ContainerDied","Data":"ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d"} Nov 03 22:50:13.390577 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.390451 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk" event={"ID":"30bba430-98f2-4ab6-a07f-a942ba4ad331","Type":"ContainerDied","Data":"15b46650589769f8acac91b61da5f400e918605a8d224c83667b0afb371890d3"} Nov 03 22:50:13.399116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.399084 4898 generic.go:356] "Generic (PLEG): container finished" podID="233153b9-fe39-4082-87c1-169894343e6b" containerID="d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0" exitCode=0 Nov 03 22:50:13.399240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.399149 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" event={"ID":"233153b9-fe39-4082-87c1-169894343e6b","Type":"ContainerDied","Data":"d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0"} Nov 03 22:50:13.399240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.399191 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" event={"ID":"233153b9-fe39-4082-87c1-169894343e6b","Type":"ContainerDied","Data":"7ac5b606e3efadba1724f0104ea2b432b99fbcdd7f5b9fc053178d0a6f2e914b"} Nov 03 22:50:13.399464 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.399385 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd" Nov 03 22:50:13.399524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.399498 4898 scope.go:122] "RemoveContainer" containerID="9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052" Nov 03 22:50:13.399848 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:13.399819 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052\": container with ID starting with 9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052 not found: ID does not exist" containerID="9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052" Nov 03 22:50:13.399962 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.399855 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052"} err="failed to get container status \"9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052\": rpc error: code = NotFound desc = could not find container \"9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052\": container with ID starting with 9ac63af5ffcf82211e5f6f71b0c255d1f37ddb7cf0a19739699f6a5cf9991052 not found: ID does not exist" Nov 03 22:50:13.399962 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.399874 4898 scope.go:122] "RemoveContainer" containerID="35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c" Nov 03 22:50:13.410586 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.410567 4898 scope.go:122] "RemoveContainer" containerID="35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c" Nov 03 22:50:13.410908 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:13.410879 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c\": container with ID starting with 35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c not found: ID does not exist" containerID="35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c" Nov 03 22:50:13.410997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.410914 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c"} err="failed to get container status \"35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c\": rpc error: code = NotFound desc = could not find container \"35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c\": container with ID starting with 35fc74cb378eb60132c3fad24e4cea318d065ea0cb6580006a5da9bc848fd33c not found: ID does not exist" Nov 03 22:50:13.410997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.410934 4898 scope.go:122] "RemoveContainer" containerID="ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d" Nov 03 22:50:13.421985 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.421962 4898 scope.go:122] "RemoveContainer" containerID="ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d" Nov 03 22:50:13.422285 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:13.422267 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d\": container with ID starting with ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d not found: ID does not exist" containerID="ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d" Nov 03 22:50:13.422364 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.422291 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d"} err="failed to get container status \"ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d\": rpc error: code = NotFound desc = could not find container \"ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d\": container with ID starting with ebb6a6cd3ec1184dccf1c40ae5e427dee102c16e35b26f3bd59d5d9d8265f02d not found: ID does not exist" Nov 03 22:50:13.422364 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.422313 4898 scope.go:122] "RemoveContainer" containerID="d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0" Nov 03 22:50:13.435480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.435454 4898 scope.go:122] "RemoveContainer" containerID="d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0" Nov 03 22:50:13.435874 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:13.435815 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0\": container with ID starting with d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0 not found: ID does not exist" containerID="d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0" Nov 03 22:50:13.435994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.435868 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0"} err="failed to get container status \"d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0\": rpc error: code = NotFound desc = could not find container \"d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0\": container with ID starting with d5cbbda705755e960caae4096f7343880423726e33aaa70cf812d5547e52e6b0 not found: ID does not exist" Nov 03 22:50:13.595612 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.595554 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg"] Nov 03 22:50:13.599115 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.599090 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-bpdgg"] Nov 03 22:50:13.732987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.732947 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.735385 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.735360 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.762200 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.762168 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk"] Nov 03 22:50:13.765671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.765648 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5z5wk"] Nov 03 22:50:13.876315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.876169 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5"] Nov 03 22:50:13.908652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.880339 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-vpkm5"] Nov 03 22:50:13.936282 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.936250 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:13.999542 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:13.998896 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd"] Nov 03 22:50:14.009843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.006326 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-72tpd"] Nov 03 22:50:14.315860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.315825 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx"] Nov 03 22:50:14.316544 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:50:14.316514 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc960bcf8_805c_4f86_a398_b476ad90ea44.slice/crio-d97ce3f67c2747a046459248529a1d3ed69c90af0a3f5af585d4e241832f2392 WatchSource:0}: Error finding container d97ce3f67c2747a046459248529a1d3ed69c90af0a3f5af585d4e241832f2392: Status 404 returned error can't find the container with id d97ce3f67c2747a046459248529a1d3ed69c90af0a3f5af585d4e241832f2392 Nov 03 22:50:14.318020 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.318001 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:50:14.411036 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.410998 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" event={"ID":"c960bcf8-805c-4f86-a398-b476ad90ea44","Type":"ContainerStarted","Data":"d97ce3f67c2747a046459248529a1d3ed69c90af0a3f5af585d4e241832f2392"} Nov 03 22:50:14.911440 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.911405 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233153b9-fe39-4082-87c1-169894343e6b" path="/var/lib/kubelet/pods/233153b9-fe39-4082-87c1-169894343e6b/volumes" Nov 03 22:50:14.912247 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.912223 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bba430-98f2-4ab6-a07f-a942ba4ad331" path="/var/lib/kubelet/pods/30bba430-98f2-4ab6-a07f-a942ba4ad331/volumes" Nov 03 22:50:14.912915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.912894 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b96c70-3f89-4389-8e41-8f79ff7e3493" path="/var/lib/kubelet/pods/62b96c70-3f89-4389-8e41-8f79ff7e3493/volumes" Nov 03 22:50:14.913928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:14.913903 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67caae8-c0f7-432a-bc11-7e72cb5750c4" path="/var/lib/kubelet/pods/d67caae8-c0f7-432a-bc11-7e72cb5750c4/volumes" Nov 03 22:50:15.420392 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:15.420351 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" event={"ID":"c960bcf8-805c-4f86-a398-b476ad90ea44","Type":"ContainerStarted","Data":"1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe"} Nov 03 22:50:15.420556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:15.420413 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:15.559811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:15.559685 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" podStartSLOduration=2.695863143 podStartE2EDuration="3.559665296s" totalImagesPullingTime="863.802153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:15.557913715 +0000 UTC m=+1658.770682795" watchObservedRunningTime="2026-11-03 22:50:15.559665296 +0000 UTC m=+1658.772434347" Nov 03 22:50:19.835382 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:19.835302 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:50:21.468173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:21.468139 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:50:27.506656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.506621 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md"] Nov 03 22:50:27.511749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.511724 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.566942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.566910 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md"] Nov 03 22:50:27.611560 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.611529 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-6a3169-serving-cert\"" Nov 03 22:50:27.673719 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.673677 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.674022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.673746 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgt8\" (UniqueName: \"kubernetes.io/projected/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-kube-api-access-qmgt8\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.674022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.673905 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.674022 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.673964 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.775222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.775140 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.775222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.775202 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgt8\" (UniqueName: \"kubernetes.io/projected/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-kube-api-access-qmgt8\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.775453 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.775280 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.775453 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.775339 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.776154 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.776120 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.777941 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.777912 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.778080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.777922 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:27.837648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:27.837614 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgt8\" (UniqueName: \"kubernetes.io/projected/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-kube-api-access-qmgt8\") pod \"maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:28.128755 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:28.128724 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:28.513847 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:28.513820 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md"] Nov 03 22:50:28.514471 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:50:28.514451 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4602fcaa_dbe8_4fd5_b02d_6326acb9d22b.slice/crio-d90d25085cca44c7626cdbfcaddef1e09aa1c7e52ead32a1f767ee1dd8601a18 WatchSource:0}: Error finding container d90d25085cca44c7626cdbfcaddef1e09aa1c7e52ead32a1f767ee1dd8601a18: Status 404 returned error can't find the container with id d90d25085cca44c7626cdbfcaddef1e09aa1c7e52ead32a1f767ee1dd8601a18 Nov 03 22:50:28.541456 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:28.541416 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" event={"ID":"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b","Type":"ContainerStarted","Data":"d90d25085cca44c7626cdbfcaddef1e09aa1c7e52ead32a1f767ee1dd8601a18"} Nov 03 22:50:29.552931 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:29.552891 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" event={"ID":"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b","Type":"ContainerStarted","Data":"39fc664c8c197b575246d825ee4f502fcd34d5481c979925abafea324933bad9"} Nov 03 22:50:29.554060 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:29.554038 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:29.696815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:29.695143 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" podStartSLOduration=1.7585378120000001 podStartE2EDuration="2.695105099s" totalImagesPullingTime="936.567287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:29.688976943 +0000 UTC m=+1672.901745987" watchObservedRunningTime="2026-11-03 22:50:29.695105099 +0000 UTC m=+1672.907874153" Nov 03 22:50:30.483451 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.483419 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5"] Nov 03 22:50:30.483649 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.483605 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" containerID="cri-o://3adb356a35cbedbc720ebc4757f8e59399ad614bfc1e5f2b6ffe9a5a3a5bf1a4" gracePeriod=30 Nov 03 22:50:30.533671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.533638 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p"] Nov 03 22:50:30.533938 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.533909 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" containerID="cri-o://0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57" gracePeriod=30 Nov 03 22:50:30.538536 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.538503 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n"] Nov 03 22:50:30.538739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.538711 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" containerID="cri-o://ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c" gracePeriod=30 Nov 03 22:50:30.585656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.585620 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5"] Nov 03 22:50:30.587408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:30.587327 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" containerID="cri-o://c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc" gracePeriod=30 Nov 03 22:50:32.051118 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.051078 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47"] Nov 03 22:50:32.051616 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.051350 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" podUID="21ad6350-f4b1-4479-b30a-72c77d8737ac" containerName="maas-api" containerID="cri-o://071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602" gracePeriod=30 Nov 03 22:50:32.525425 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.525096 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:50:32.603434 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.603398 4898 generic.go:356] "Generic (PLEG): container finished" podID="21ad6350-f4b1-4479-b30a-72c77d8737ac" containerID="071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602" exitCode=0 Nov 03 22:50:32.603847 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.603812 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" event={"ID":"21ad6350-f4b1-4479-b30a-72c77d8737ac","Type":"ContainerDied","Data":"071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602"} Nov 03 22:50:32.603955 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.603898 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" event={"ID":"21ad6350-f4b1-4479-b30a-72c77d8737ac","Type":"ContainerDied","Data":"af7e3a6a4041cacb877d02df87d89ff754d6dcf44ec0cf15984af2408f90054f"} Nov 03 22:50:32.603955 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.603949 4898 scope.go:122] "RemoveContainer" containerID="071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602" Nov 03 22:50:32.604202 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.604177 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" Nov 03 22:50:32.624832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.622640 4898 scope.go:122] "RemoveContainer" containerID="071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602" Nov 03 22:50:32.624832 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:32.623098 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602\": container with ID starting with 071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602 not found: ID does not exist" containerID="071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602" Nov 03 22:50:32.624832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.623143 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602"} err="failed to get container status \"071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602\": rpc error: code = NotFound desc = could not find container \"071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602\": container with ID starting with 071b6bfff890c6673f1d4ab52476ece2f29cd67e013f65460ef094da792c7602 not found: ID does not exist" Nov 03 22:50:32.633167 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.633114 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-metrics-certs\") pod \"21ad6350-f4b1-4479-b30a-72c77d8737ac\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " Nov 03 22:50:32.633336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.633244 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ad6350-f4b1-4479-b30a-72c77d8737ac-kube-api-access-vpcxr\" (UniqueName: \"kubernetes.io/projected/21ad6350-f4b1-4479-b30a-72c77d8737ac-kube-api-access-vpcxr\") pod \"21ad6350-f4b1-4479-b30a-72c77d8737ac\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " Nov 03 22:50:32.633404 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.633335 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ad6350-f4b1-4479-b30a-72c77d8737ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad6350-f4b1-4479-b30a-72c77d8737ac-openshift-service-ca\") pod \"21ad6350-f4b1-4479-b30a-72c77d8737ac\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " Nov 03 22:50:32.633460 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.633411 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls\") pod \"21ad6350-f4b1-4479-b30a-72c77d8737ac\" (UID: \"21ad6350-f4b1-4479-b30a-72c77d8737ac\") " Nov 03 22:50:32.634192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.634164 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ad6350-f4b1-4479-b30a-72c77d8737ac-openshift-service-ca" pod "21ad6350-f4b1-4479-b30a-72c77d8737ac" (UID: "21ad6350-f4b1-4479-b30a-72c77d8737ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:32.635958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.635930 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls" pod "21ad6350-f4b1-4479-b30a-72c77d8737ac" (UID: "21ad6350-f4b1-4479-b30a-72c77d8737ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:32.636104 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.636086 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ad6350-f4b1-4479-b30a-72c77d8737ac-kube-api-access-vpcxr" pod "21ad6350-f4b1-4479-b30a-72c77d8737ac" (UID: "21ad6350-f4b1-4479-b30a-72c77d8737ac"). InnerVolumeSpecName "kube-api-access-vpcxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:32.639442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.639417 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-metrics-certs" pod "21ad6350-f4b1-4479-b30a-72c77d8737ac" (UID: "21ad6350-f4b1-4479-b30a-72c77d8737ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:32.739310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.739229 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:32.739310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.739261 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpcxr\" (UniqueName: \"kubernetes.io/projected/21ad6350-f4b1-4479-b30a-72c77d8737ac-kube-api-access-vpcxr\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:32.739310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.739273 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad6350-f4b1-4479-b30a-72c77d8737ac-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:32.739310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:32.739284 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad6350-f4b1-4479-b30a-72c77d8737ac-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:33.036523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:33.036425 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47"] Nov 03 22:50:33.094869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:33.089075 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47"] Nov 03 22:50:34.908549 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:34.908506 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ad6350-f4b1-4479-b30a-72c77d8737ac" path="/var/lib/kubelet/pods/21ad6350-f4b1-4479-b30a-72c77d8737ac/volumes" Nov 03 22:50:35.073059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.073017 4898 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:35.073215 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.073085 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:35.380025 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.379961 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d18a5b05-767c5996f8-m4l47" podUID="21ad6350-f4b1-4479-b30a-72c77d8737ac" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.95:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:50:35.640632 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.640602 4898 generic.go:356] "Generic (PLEG): container finished" podID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerID="3adb356a35cbedbc720ebc4757f8e59399ad614bfc1e5f2b6ffe9a5a3a5bf1a4" exitCode=0 Nov 03 22:50:35.640753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.640675 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" event={"ID":"12f07358-7c7d-45f1-ac0a-7884d16bfca4","Type":"ContainerDied","Data":"3adb356a35cbedbc720ebc4757f8e59399ad614bfc1e5f2b6ffe9a5a3a5bf1a4"} Nov 03 22:50:35.823996 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.823968 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:50:35.896609 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.896525 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-podinfo\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.896609 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.896590 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-certs\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.896837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.896690 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istiod-ca-cert\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.896837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.896726 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-kube-api-access-47hbf\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-kube-api-access-47hbf\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.896837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.896768 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-token\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-token\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.897010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.896879 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-credential-socket\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.897010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.896915 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-data\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.897120 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.897015 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-socket\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.897120 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.897061 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-envoy\") pod \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\" (UID: \"12f07358-7c7d-45f1-ac0a-7884d16bfca4\") " Nov 03 22:50:35.898978 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.898943 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-credential-socket" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.899503 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.899459 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-certs" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.899963 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.899936 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istiod-ca-cert" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:35.900047 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.900003 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-socket" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.900120 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.900076 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-data" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.902080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.902053 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-token" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:35.905754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.905730 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-podinfo" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:35.907071 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.907050 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-envoy" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.926035 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.926007 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-kube-api-access-47hbf" pod "12f07358-7c7d-45f1-ac0a-7884d16bfca4" (UID: "12f07358-7c7d-45f1-ac0a-7884d16bfca4"). InnerVolumeSpecName "kube-api-access-47hbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:35.963757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.963727 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:50:35.971967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.971946 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:50:35.985261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.985233 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:50:35.998763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.998733 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-certs\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.998881 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.998841 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-envoy\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.998924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.998888 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-token\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-token\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.998972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.998925 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00993844-beef-4932-b545-d4f3dc410c15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00993844-beef-4932-b545-d4f3dc410c15-istio-podinfo\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.998972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.998960 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-credential-socket\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.999036 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999012 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-credential-socket\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.999074 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999045 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-certs" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.999074 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999065 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-data\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.999160 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999098 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-kube-api-access-m2lpj\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-kube-api-access-m2lpj\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.999160 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999124 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-data\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.999234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999196 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-certs\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.999269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999230 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-socket\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.999311 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999296 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istiod-ca-cert\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.999371 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999348 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-credential-socket" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.999429 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999360 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00993844-beef-4932-b545-d4f3dc410c15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00993844-beef-4932-b545-d4f3dc410c15-istiod-ca-cert\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.999485 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999448 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-podinfo\") pod \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\" (UID: \"a638b3fc-f541-4ab3-b0ad-7fd36574b367\") " Nov 03 22:50:35.999485 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999470 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-envoy\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.999603 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999510 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-socket\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.999664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999599 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-kube-api-access-fnf86\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-kube-api-access-fnf86\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:35.999664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:35.999646 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-istio-token\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-istio-token\") pod \"00993844-beef-4932-b545-d4f3dc410c15\" (UID: \"00993844-beef-4932-b545-d4f3dc410c15\") " Nov 03 22:50:36.000714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.000183 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-data" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.000714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.000203 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-socket" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.000714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.000338 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-certs" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.000714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.000536 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-data" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.000714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.000547 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istiod-ca-cert" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:36.001047 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.000986 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-socket" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.001441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.001167 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-credential-socket" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.002001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.001723 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00993844-beef-4932-b545-d4f3dc410c15-istiod-ca-cert" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:36.002096 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002061 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-envoy" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.002153 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002102 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-token" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:36.002500 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002473 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002507 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002521 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002534 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002548 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002566 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002580 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002626 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00993844-beef-4932-b545-d4f3dc410c15-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002642 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002658 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002673 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002688 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002705 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47hbf\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-kube-api-access-47hbf\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002724 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002739 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002754 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002769 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002801 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002815 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002831 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f07358-7c7d-45f1-ac0a-7884d16bfca4-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.002894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.002847 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a638b3fc-f541-4ab3-b0ad-7fd36574b367-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.004256 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.003470 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-envoy" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.004256 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.003533 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-kube-api-access-m2lpj" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "kube-api-access-m2lpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:36.004579 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.004555 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-kube-api-access-fnf86" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "kube-api-access-fnf86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:36.005041 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.005011 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00993844-beef-4932-b545-d4f3dc410c15-istio-podinfo" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:36.006062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.006032 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-podinfo" pod "a638b3fc-f541-4ab3-b0ad-7fd36574b367" (UID: "a638b3fc-f541-4ab3-b0ad-7fd36574b367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:36.007096 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.007071 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-istio-token" pod "00993844-beef-4932-b545-d4f3dc410c15" (UID: "00993844-beef-4932-b545-d4f3dc410c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:36.103452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103417 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istiod-ca-cert\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.103635 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103463 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-socket\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.103635 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103621 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-token\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-token\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.103737 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103665 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-podinfo\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.103801 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103772 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-certs\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.103850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103783 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-socket" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.103900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103873 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-kube-api-access-zb9bp\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-kube-api-access-zb9bp\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.103941 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103919 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-credential-socket\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.103941 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.103880 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istiod-ca-cert" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:36.104040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104001 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-envoy\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.104040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104031 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-data\") pod \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\" (UID: \"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e\") " Nov 03 22:50:36.104148 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104069 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-certs" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.104424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104159 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-credential-socket" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.104424 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104407 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-data" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104680 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104708 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104723 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104737 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00993844-beef-4932-b545-d4f3dc410c15-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104754 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104767 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104784 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2lpj\" (UniqueName: \"kubernetes.io/projected/a638b3fc-f541-4ab3-b0ad-7fd36574b367-kube-api-access-m2lpj\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104825 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a638b3fc-f541-4ab3-b0ad-7fd36574b367-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104839 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00993844-beef-4932-b545-d4f3dc410c15-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104852 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnf86\" (UniqueName: \"kubernetes.io/projected/00993844-beef-4932-b545-d4f3dc410c15-kube-api-access-fnf86\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.104890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.104866 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.105982 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.105957 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-podinfo" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:36.106186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.106164 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-kube-api-access-zb9bp" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "kube-api-access-zb9bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:36.106186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.106174 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-token" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:36.106341 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.106326 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-envoy" pod "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" (UID: "9435ac03-5c16-4fe5-ba56-a5934c4f5a8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:36.206525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.206444 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.206525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.206476 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.206525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.206487 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9bp\" (UniqueName: \"kubernetes.io/projected/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-kube-api-access-zb9bp\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.206525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.206498 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:36.592111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.592083 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:50:36.653473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.653440 4898 generic.go:356] "Generic (PLEG): container finished" podID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerID="0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57" exitCode=0 Nov 03 22:50:36.653671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.653493 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" Nov 03 22:50:36.653671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.653522 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" event={"ID":"a638b3fc-f541-4ab3-b0ad-7fd36574b367","Type":"ContainerDied","Data":"0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57"} Nov 03 22:50:36.653671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.653588 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p" event={"ID":"a638b3fc-f541-4ab3-b0ad-7fd36574b367","Type":"ContainerDied","Data":"fd4141a636af3c823cc5f859122a3237610511a6a6af597ea44b48780accdb7a"} Nov 03 22:50:36.653671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.653620 4898 scope.go:122] "RemoveContainer" containerID="0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57" Nov 03 22:50:36.655382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.655296 4898 generic.go:356] "Generic (PLEG): container finished" podID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerID="c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc" exitCode=0 Nov 03 22:50:36.655382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.655365 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" Nov 03 22:50:36.655511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.655407 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" event={"ID":"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e","Type":"ContainerDied","Data":"c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc"} Nov 03 22:50:36.655511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.655490 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5" event={"ID":"9435ac03-5c16-4fe5-ba56-a5934c4f5a8e","Type":"ContainerDied","Data":"d08451f6f267b9c6bf1050cb527e8fc6f6bd134b0febf98d30a86cc679255fd1"} Nov 03 22:50:36.657253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.657228 4898 generic.go:356] "Generic (PLEG): container finished" podID="00993844-beef-4932-b545-d4f3dc410c15" containerID="ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c" exitCode=0 Nov 03 22:50:36.657253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.657266 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" Nov 03 22:50:36.657253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.657298 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" event={"ID":"00993844-beef-4932-b545-d4f3dc410c15","Type":"ContainerDied","Data":"ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c"} Nov 03 22:50:36.657253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.657382 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n" event={"ID":"00993844-beef-4932-b545-d4f3dc410c15","Type":"ContainerDied","Data":"03eaf89d3be988be359dcdf92b28119ac467fc9861bc629c79aeb563838a4b8b"} Nov 03 22:50:36.659416 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.659393 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" event={"ID":"12f07358-7c7d-45f1-ac0a-7884d16bfca4","Type":"ContainerDied","Data":"4ed7b9b07aaa252c25c0e0a172cfda75b775fb01664c8267fd74c612fa8b86eb"} Nov 03 22:50:36.659684 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.659479 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5" Nov 03 22:50:36.670229 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.669770 4898 scope.go:122] "RemoveContainer" containerID="0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57" Nov 03 22:50:36.670229 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:36.670209 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57\": container with ID starting with 0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57 not found: ID does not exist" containerID="0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57" Nov 03 22:50:36.670590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.670242 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57"} err="failed to get container status \"0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57\": rpc error: code = NotFound desc = could not find container \"0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57\": container with ID starting with 0f9d8ea854f0baed113faefb6f122624f6a566f941c0e1efab531a0665a2fc57 not found: ID does not exist" Nov 03 22:50:36.670590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.670267 4898 scope.go:122] "RemoveContainer" containerID="c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc" Nov 03 22:50:36.682348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.682328 4898 scope.go:122] "RemoveContainer" containerID="c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc" Nov 03 22:50:36.682644 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:36.682626 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc\": container with ID starting with c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc not found: ID does not exist" containerID="c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc" Nov 03 22:50:36.682724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.682653 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc"} err="failed to get container status \"c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc\": rpc error: code = NotFound desc = could not find container \"c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc\": container with ID starting with c3cdd2476774e5fa8fd77fc5ac637639069085e74047d690df9e4a6f61c7a4dc not found: ID does not exist" Nov 03 22:50:36.682724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.682672 4898 scope.go:122] "RemoveContainer" containerID="ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c" Nov 03 22:50:36.697484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.697463 4898 scope.go:122] "RemoveContainer" containerID="ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c" Nov 03 22:50:36.697956 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:36.697883 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c\": container with ID starting with ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c not found: ID does not exist" containerID="ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c" Nov 03 22:50:36.697956 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.697935 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c"} err="failed to get container status \"ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c\": rpc error: code = NotFound desc = could not find container \"ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c\": container with ID starting with ecebf5d6cea230692fb8657c56f6ed372b4e7f1e20b67d3ee257d05356c2694c not found: ID does not exist" Nov 03 22:50:36.698197 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.697962 4898 scope.go:122] "RemoveContainer" containerID="3adb356a35cbedbc720ebc4757f8e59399ad614bfc1e5f2b6ffe9a5a3a5bf1a4" Nov 03 22:50:36.833058 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.833027 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5"] Nov 03 22:50:36.871768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.871737 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r69n5"] Nov 03 22:50:36.907354 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.907326 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" path="/var/lib/kubelet/pods/9435ac03-5c16-4fe5-ba56-a5934c4f5a8e/volumes" Nov 03 22:50:36.984752 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.984719 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5"] Nov 03 22:50:36.989634 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:36.989605 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-ff7t5"] Nov 03 22:50:37.097446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:37.097370 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p"] Nov 03 22:50:37.100149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:37.100125 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-8kw9p"] Nov 03 22:50:37.214511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:37.214432 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n"] Nov 03 22:50:37.217519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:37.217492 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-dn59n"] Nov 03 22:50:38.908125 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:38.908092 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00993844-beef-4932-b545-d4f3dc410c15" path="/var/lib/kubelet/pods/00993844-beef-4932-b545-d4f3dc410c15/volumes" Nov 03 22:50:38.908590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:38.908580 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" path="/var/lib/kubelet/pods/12f07358-7c7d-45f1-ac0a-7884d16bfca4/volumes" Nov 03 22:50:38.909069 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:38.909046 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" path="/var/lib/kubelet/pods/a638b3fc-f541-4ab3-b0ad-7fd36574b367/volumes" Nov 03 22:50:43.486499 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.486466 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm"] Nov 03 22:50:43.487097 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487082 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" Nov 03 22:50:43.487143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487098 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" Nov 03 22:50:43.487143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487112 4898 cpu_manager.go:414] "containerMap: removing container" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" Nov 03 22:50:43.487143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487117 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" Nov 03 22:50:43.487143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487132 4898 cpu_manager.go:414] "containerMap: removing container" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" Nov 03 22:50:43.487143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487137 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487147 4898 cpu_manager.go:414] "containerMap: removing container" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487152 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487171 4898 cpu_manager.go:414] "containerMap: removing container" podUID="21ad6350-f4b1-4479-b30a-72c77d8737ac" containerName="maas-api" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487175 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ad6350-f4b1-4479-b30a-72c77d8737ac" containerName="maas-api" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487261 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a638b3fc-f541-4ab3-b0ad-7fd36574b367" containerName="istio-proxy" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487271 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9435ac03-5c16-4fe5-ba56-a5934c4f5a8e" containerName="istio-proxy" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487280 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f07358-7c7d-45f1-ac0a-7884d16bfca4" containerName="istio-proxy" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487287 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="00993844-beef-4932-b545-d4f3dc410c15" containerName="istio-proxy" Nov 03 22:50:43.487298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.487295 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ad6350-f4b1-4479-b30a-72c77d8737ac" containerName="maas-api" Nov 03 22:50:43.492992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.492971 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.559988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.559956 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm"] Nov 03 22:50:43.605389 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.605357 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2627c5-openshift-default-dockercfg-9qtts\"" Nov 03 22:50:43.641859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.639673 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istiod-ca-cert\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.641859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.639756 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42t5x\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-kube-api-access-42t5x\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.641859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.639917 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-data\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.641859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.639954 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-token\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.645375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.644681 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-envoy\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.645375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.644834 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-socket\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.645375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.644936 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-certs\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.645375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.644980 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-podinfo\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.645375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.645120 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-credential-socket\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.746711 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.746617 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-credential-socket\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.746711 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.746674 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istiod-ca-cert\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.746994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.746712 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42t5x\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-kube-api-access-42t5x\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747137 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747112 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-credential-socket\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747409 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747382 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-data\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747449 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-data\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747487 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-token\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747538 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-envoy\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747449 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istiod-ca-cert\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747703 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747668 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-socket\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747760 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-certs\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.747850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747816 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-podinfo\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.748001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.747977 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-socket\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.748273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.748246 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-certs\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.750413 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.750392 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-envoy\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.750981 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.750958 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-podinfo\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.809755 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.809713 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-token\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:43.809947 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:43.809718 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42t5x\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-kube-api-access-42t5x\") pod \"e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:44.108089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.108055 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:44.550358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.550268 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm"] Nov 03 22:50:44.588701 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:50:44.588657 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b55968_7e4c_4508_b0b4_a64ad6eeb74d.slice/crio-170e41aee369f156d75a8b8333cb2eea737ff9c61ad65f04ec30d6a35b605b8c WatchSource:0}: Error finding container 170e41aee369f156d75a8b8333cb2eea737ff9c61ad65f04ec30d6a35b605b8c: Status 404 returned error can't find the container with id 170e41aee369f156d75a8b8333cb2eea737ff9c61ad65f04ec30d6a35b605b8c Nov 03 22:50:44.591300 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.591264 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:44.591382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.591339 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:44.591382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.591368 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:44.723859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.723818 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" event={"ID":"08b55968-7e4c-4508-b0b4-a64ad6eeb74d","Type":"ContainerStarted","Data":"922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e"} Nov 03 22:50:44.724060 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.723878 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" event={"ID":"08b55968-7e4c-4508-b0b4-a64ad6eeb74d","Type":"ContainerStarted","Data":"170e41aee369f156d75a8b8333cb2eea737ff9c61ad65f04ec30d6a35b605b8c"} Nov 03 22:50:44.895086 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:44.895035 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podStartSLOduration=1.89502055 podStartE2EDuration="1.89502055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:44.89346829 +0000 UTC m=+1688.106237340" watchObservedRunningTime="2026-11-03 22:50:44.89502055 +0000 UTC m=+1688.107789601" Nov 03 22:50:45.108370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:45.108282 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:45.115148 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:45.115118 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:45.730502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:45.730472 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:45.731485 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:45.731466 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:50:49.834932 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:49.834891 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:50:52.083816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.081921 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl"] Nov 03 22:50:52.133815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.120096 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.220812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.206191 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2420f6-openshift-default-dockercfg-9mz98\"" Nov 03 22:50:52.220812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.211806 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl"] Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.224927 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b996d093-0a87-4e45-afc4-4488b00f78da-istiod-ca-cert\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225050 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-socket\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225119 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-certs\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225145 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-data\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225194 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-credential-socket\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225250 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b996d093-0a87-4e45-afc4-4488b00f78da-istio-podinfo\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225287 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-istio-token\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225316 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmrc\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-kube-api-access-xkmrc\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.246129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.225450 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-envoy\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.326729 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.326690 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-envoy\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.327487 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.327462 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b996d093-0a87-4e45-afc4-4488b00f78da-istiod-ca-cert\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.328398 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.328379 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-socket\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.328565 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.328550 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-certs\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.328666 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.328651 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-data\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.329041 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.329023 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-data\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.329673 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.328315 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b996d093-0a87-4e45-afc4-4488b00f78da-istiod-ca-cert\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.329822 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.329295 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-credential-socket\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.329994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.329976 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b996d093-0a87-4e45-afc4-4488b00f78da-istio-podinfo\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.330592 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.329578 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-credential-socket\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.330592 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.329621 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-certs\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.330592 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.329386 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-socket\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.330866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.330846 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-istio-token\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.331358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.331337 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmrc\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-kube-api-access-xkmrc\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.332952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.332931 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b996d093-0a87-4e45-afc4-4488b00f78da-istio-podinfo\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.338708 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.338657 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-envoy\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.423583 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.423552 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-istio-token\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.430459 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.430427 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmrc\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-kube-api-access-xkmrc\") pod \"e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:52.580817 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:52.580043 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:53.327717 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:53.327688 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl"] Nov 03 22:50:53.328191 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:50:53.328160 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb996d093_0a87_4e45_afc4_4488b00f78da.slice/crio-3f17e1f8ae4b7e27518d555bc519c11b69fbc87d05cb62e49089817c6a7a58b9 WatchSource:0}: Error finding container 3f17e1f8ae4b7e27518d555bc519c11b69fbc87d05cb62e49089817c6a7a58b9: Status 404 returned error can't find the container with id 3f17e1f8ae4b7e27518d555bc519c11b69fbc87d05cb62e49089817c6a7a58b9 Nov 03 22:50:53.336521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:53.336483 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:53.336638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:53.336577 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:53.336701 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:53.336642 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:53.910218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:53.910180 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" event={"ID":"b996d093-0a87-4e45-afc4-4488b00f78da","Type":"ContainerStarted","Data":"01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08"} Nov 03 22:50:53.910426 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:53.910240 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" event={"ID":"b996d093-0a87-4e45-afc4-4488b00f78da","Type":"ContainerStarted","Data":"3f17e1f8ae4b7e27518d555bc519c11b69fbc87d05cb62e49089817c6a7a58b9"} Nov 03 22:50:54.081815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.081748 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podStartSLOduration=2.081733115 podStartE2EDuration="2.081733115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:54.080472878 +0000 UTC m=+1697.293241948" watchObservedRunningTime="2026-11-03 22:50:54.081733115 +0000 UTC m=+1697.294502166" Nov 03 22:50:54.412356 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.412308 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl"] Nov 03 22:50:54.417326 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.417300 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.471815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.470921 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0c21c7-serving-cert\"" Nov 03 22:50:54.476040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.476009 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl"] Nov 03 22:50:54.581239 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.581209 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:54.581917 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.581894 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwvk\" (UniqueName: \"kubernetes.io/projected/220db423-ef85-4549-a07f-353c37c06192-kube-api-access-grwvk\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.582044 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.581980 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-metrics-certs\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.582136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.582109 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220db423-ef85-4549-a07f-353c37c06192-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.582246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.582223 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.586939 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.586918 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:54.683261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.683168 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.683438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.683347 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwvk\" (UniqueName: \"kubernetes.io/projected/220db423-ef85-4549-a07f-353c37c06192-kube-api-access-grwvk\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.683438 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:54.683364 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-serving-cert: secret "maas-api-e2e-api-a-0c21c7-serving-cert" not found Nov 03 22:50:54.683543 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:50:54.683451 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls podName:220db423-ef85-4549-a07f-353c37c06192 nodeName:}" failed. No retries permitted until 2026-11-03 22:50:55.183430257 +0000 UTC m=+1698.396199303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls") pod "maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" (UID: "220db423-ef85-4549-a07f-353c37c06192") : secret "maas-api-e2e-api-a-0c21c7-serving-cert" not found Nov 03 22:50:54.683543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.683491 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-metrics-certs\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.683653 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.683565 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220db423-ef85-4549-a07f-353c37c06192-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.684484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.684459 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220db423-ef85-4549-a07f-353c37c06192-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.686641 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.686619 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-metrics-certs\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.745077 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.745041 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwvk\" (UniqueName: \"kubernetes.io/projected/220db423-ef85-4549-a07f-353c37c06192-kube-api-access-grwvk\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:54.916432 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.916398 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:54.917482 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:54.917457 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:50:55.190419 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:55.190384 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:55.192877 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:55.192856 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls\") pod \"maas-api-e2e-api-a-0c21c7-f5569d595-z59wl\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:55.332266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:55.332225 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:55.756594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:55.756568 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl"] Nov 03 22:50:55.923997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:55.923957 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" event={"ID":"220db423-ef85-4549-a07f-353c37c06192","Type":"ContainerStarted","Data":"b80d1600bfaaa1d3b5b48d86d48f490b8db6734a1d3858120ce6e3083134368e"} Nov 03 22:50:56.935946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:56.935847 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" event={"ID":"220db423-ef85-4549-a07f-353c37c06192","Type":"ContainerStarted","Data":"69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659"} Nov 03 22:50:56.935946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:56.935916 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:50:57.095402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:50:57.095331 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" podStartSLOduration=2.26193562 podStartE2EDuration="3.095315906s" totalImagesPullingTime="833.380286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:57.093565631 +0000 UTC m=+1700.306334700" watchObservedRunningTime="2026-11-03 22:50:57.095315906 +0000 UTC m=+1700.308085002" Nov 03 22:51:02.949548 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:02.949519 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:51:19.839654 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:51:19.839571 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:51:20.032133 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.032100 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47"] Nov 03 22:51:20.037711 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.037686 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.096095 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.093150 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-922795-openshift-default-dockercfg-9r6md\"" Nov 03 22:51:20.150275 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.150234 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47"] Nov 03 22:51:20.151411 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.151380 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-socket\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.155922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.151672 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-envoy\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.156753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.156729 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-istio-token\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.156983 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.156966 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba948794-2afd-408b-bd73-b7f1d520f746-istiod-ca-cert\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.157171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.157156 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba948794-2afd-408b-bd73-b7f1d520f746-istio-podinfo\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.157377 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.157360 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-data\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.157545 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.157529 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wkz\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-kube-api-access-r9wkz\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.157695 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.157680 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-certs\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.167372 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.167347 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-credential-socket\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.300886 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.276767 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-istio-token\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.301190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.301157 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba948794-2afd-408b-bd73-b7f1d520f746-istiod-ca-cert\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.311725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.301349 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba948794-2afd-408b-bd73-b7f1d520f746-istio-podinfo\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.310685 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba948794-2afd-408b-bd73-b7f1d520f746-istiod-ca-cert\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.313433 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-data\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.313499 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wkz\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-kube-api-access-r9wkz\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.313533 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-certs\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.313578 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-credential-socket\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.313715 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-socket\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.313759 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-data\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.313779 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-envoy\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.314524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.314238 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-certs\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.315175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.314562 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-credential-socket\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.315175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.314655 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-socket\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.317127 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.317076 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba948794-2afd-408b-bd73-b7f1d520f746-istio-podinfo\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.317489 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.317436 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-envoy\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.372285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.372197 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-istio-token\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.376062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.376034 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wkz\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-kube-api-access-r9wkz\") pod \"e2e-api-a-922795-openshift-default-59b46784df-6bc47\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:20.654132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:20.654045 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:21.094650 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:21.094616 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47"] Nov 03 22:51:21.095321 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:51:21.095297 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba948794_2afd_408b_bd73_b7f1d520f746.slice/crio-d080ec3f121714fe9e0eb881a8f2acccc40b1df434a46b3ba7df98b9850c7236 WatchSource:0}: Error finding container d080ec3f121714fe9e0eb881a8f2acccc40b1df434a46b3ba7df98b9850c7236: Status 404 returned error can't find the container with id d080ec3f121714fe9e0eb881a8f2acccc40b1df434a46b3ba7df98b9850c7236 Nov 03 22:51:21.098037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:21.098004 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:21.098122 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:21.098067 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:21.098122 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:21.098094 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:21.112301 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:21.112271 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" event={"ID":"ba948794-2afd-408b-bd73-b7f1d520f746","Type":"ContainerStarted","Data":"d080ec3f121714fe9e0eb881a8f2acccc40b1df434a46b3ba7df98b9850c7236"} Nov 03 22:51:22.119754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:22.119708 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" event={"ID":"ba948794-2afd-408b-bd73-b7f1d520f746","Type":"ContainerStarted","Data":"014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018"} Nov 03 22:51:22.293332 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:22.293277 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podStartSLOduration=3.293263501 podStartE2EDuration="3.293263501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:22.29013404 +0000 UTC m=+1725.502903108" watchObservedRunningTime="2026-11-03 22:51:22.293263501 +0000 UTC m=+1725.506032551" Nov 03 22:51:22.654580 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:22.654539 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:22.663111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:22.663079 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:23.126078 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:23.126046 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:23.126976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:23.126952 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:51:32.649268 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.647982 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk"] Nov 03 22:51:32.653940 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.653911 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.727810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.722121 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-65913e-openshift-default-dockercfg-w4x97\"" Nov 03 22:51:32.728888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.728855 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4t4\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-kube-api-access-gl4t4\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.740813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.729093 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-credential-socket\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.741112 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.741085 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-certs\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.741254 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.741239 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-socket\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.741390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.741370 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-envoy\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.748826 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.741566 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-token\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.749045 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.749020 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-data\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.772814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.749188 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-podinfo\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.773123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.773093 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952a43e1-5149-45e3-8c2e-55eec1b64d63-istiod-ca-cert\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.877138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.876528 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4t4\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-kube-api-access-gl4t4\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884064 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-credential-socket\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884191 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-certs\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884249 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-socket\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884291 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-envoy\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884373 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-token\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884405 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-data\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884461 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-podinfo\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.884495 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952a43e1-5149-45e3-8c2e-55eec1b64d63-istiod-ca-cert\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.886099 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-certs\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.886340 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-socket\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.888604 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-data\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.895908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.888858 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952a43e1-5149-45e3-8c2e-55eec1b64d63-istiod-ca-cert\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.896718 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.896691 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-credential-socket\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.904075 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.904003 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-envoy\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.914129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.914072 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-podinfo\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.919836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.919743 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk"] Nov 03 22:51:32.939696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.939660 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4t4\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-kube-api-access-gl4t4\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.953491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.953440 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-token\") pod \"e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:32.999162 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:32.999130 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:33.474221 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:33.474190 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk"] Nov 03 22:51:33.512363 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:51:33.512328 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952a43e1_5149_45e3_8c2e_55eec1b64d63.slice/crio-9538b5e53dbcb454a415deef4ad1f618a8e96a2481df698cc5daf8903fb781ed WatchSource:0}: Error finding container 9538b5e53dbcb454a415deef4ad1f618a8e96a2481df698cc5daf8903fb781ed: Status 404 returned error can't find the container with id 9538b5e53dbcb454a415deef4ad1f618a8e96a2481df698cc5daf8903fb781ed Nov 03 22:51:33.516062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:33.516029 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:33.516169 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:33.516103 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:33.516169 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:33.516134 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:34.222595 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:34.222553 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" event={"ID":"952a43e1-5149-45e3-8c2e-55eec1b64d63","Type":"ContainerStarted","Data":"b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd"} Nov 03 22:51:34.223032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:34.222616 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" event={"ID":"952a43e1-5149-45e3-8c2e-55eec1b64d63","Type":"ContainerStarted","Data":"9538b5e53dbcb454a415deef4ad1f618a8e96a2481df698cc5daf8903fb781ed"} Nov 03 22:51:34.449748 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:34.449681 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podStartSLOduration=2.449663046 podStartE2EDuration="2.449663046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:34.395219863 +0000 UTC m=+1737.607988929" watchObservedRunningTime="2026-11-03 22:51:34.449663046 +0000 UTC m=+1737.662432097" Nov 03 22:51:34.999798 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:34.999741 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:35.005742 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:35.005718 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:35.230851 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:35.230821 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:35.231924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:35.231905 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:51:47.624829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.621757 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f"] Nov 03 22:51:47.633774 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.632823 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.702809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.693011 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0189ef-openshift-default-dockercfg-2c8ld\"" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835023 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-envoy\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835103 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8cl\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-kube-api-access-5r8cl\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835130 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-podinfo\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835270 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-certs\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835309 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istiod-ca-cert\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835409 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-credential-socket\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835469 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-socket\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835491 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-data\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.835586 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-token\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.849340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.845930 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f"] Nov 03 22:51:47.937132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.937062 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-certs\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.937483 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.937463 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istiod-ca-cert\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.937666 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.937653 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-credential-socket\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.937762 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.937753 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-socket\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.937864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.937844 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-data\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.938016 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.937992 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-token\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.938110 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.938092 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-envoy\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.938174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.938154 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8cl\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-kube-api-access-5r8cl\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.938231 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.938184 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-podinfo\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.942918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.939136 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-credential-socket\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.942918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.941965 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-socket\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.942918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.942202 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-data\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.943154 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.942940 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-certs\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.943570 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.943548 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istiod-ca-cert\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.955297 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.951157 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-envoy\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:47.956679 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:47.956657 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-podinfo\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:48.007343 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:48.007293 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-token\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:48.007542 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:48.007386 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8cl\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-kube-api-access-5r8cl\") pod \"e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:48.275188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:48.275103 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:48.728241 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:48.728210 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f"] Nov 03 22:51:48.729343 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:51:48.729311 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf648795_6e86_4ba4_95a8_a9aa0bd7d4c3.slice/crio-ed6d93e4c3e0a566dff205bf0a991c8405245302c7a6faa4eb3f0f24dca343ac WatchSource:0}: Error finding container ed6d93e4c3e0a566dff205bf0a991c8405245302c7a6faa4eb3f0f24dca343ac: Status 404 returned error can't find the container with id ed6d93e4c3e0a566dff205bf0a991c8405245302c7a6faa4eb3f0f24dca343ac Nov 03 22:51:48.732610 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:48.732578 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:48.732686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:48.732645 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:48.732732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:48.732696 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:49.328265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:49.328232 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" event={"ID":"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3","Type":"ContainerStarted","Data":"81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274"} Nov 03 22:51:49.328469 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:49.328286 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" event={"ID":"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3","Type":"ContainerStarted","Data":"ed6d93e4c3e0a566dff205bf0a991c8405245302c7a6faa4eb3f0f24dca343ac"} Nov 03 22:51:49.452244 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:49.452192 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podStartSLOduration=2.452179317 podStartE2EDuration="2.452179317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:49.449344019 +0000 UTC m=+1752.662113070" watchObservedRunningTime="2026-11-03 22:51:49.452179317 +0000 UTC m=+1752.664948368" Nov 03 22:51:49.835962 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:51:49.835917 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:51:50.275628 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:50.275543 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:50.281639 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:50.281610 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:50.335200 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:50.335165 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:50.336072 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:50.336047 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:51:52.719963 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:52.719929 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9"] Nov 03 22:51:52.720580 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:52.720146 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" containerID="cri-o://3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924" gracePeriod=30 Nov 03 22:51:58.091375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.091099 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:51:58.293452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293352 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istiod-ca-cert\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293618 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293475 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-kube-api-access-z2ngt\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-kube-api-access-z2ngt\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293618 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293522 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-socket\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293618 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293612 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-podinfo\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293669 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-certs\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293773 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-credential-socket\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293838 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istiod-ca-cert" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:58.293984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293857 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-envoy\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293883 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-token\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293911 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-data\") pod \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\" (UID: \"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20\") " Nov 03 22:51:58.293984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.293964 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-socket" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:58.294303 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294089 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-certs" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:58.294303 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294108 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-credential-socket" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:58.294405 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294320 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-data" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:58.294576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294494 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.294576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294516 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.294576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294530 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.294576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294543 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.294576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.294555 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.296478 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.296454 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-envoy" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:58.298448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.298417 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-podinfo" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:58.298559 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.298462 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-token" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:58.298559 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.298527 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-kube-api-access-z2ngt" pod "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" (UID: "2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20"). InnerVolumeSpecName "kube-api-access-z2ngt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:58.401124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.401086 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.401124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.401125 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.401124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.401140 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ngt\" (UniqueName: \"kubernetes.io/projected/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-kube-api-access-z2ngt\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.401395 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.401154 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.417208 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.417164 4898 generic.go:356] "Generic (PLEG): container finished" podID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerID="3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924" exitCode=0 Nov 03 22:51:58.417416 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.417355 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" event={"ID":"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20","Type":"ContainerDied","Data":"3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924"} Nov 03 22:51:58.417416 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.417410 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" event={"ID":"2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20","Type":"ContainerDied","Data":"4dbed896cffb6d2daf047b7c4dcd5a3c5f64bdec8abe07cbc8e1da8a86b02891"} Nov 03 22:51:58.417776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.417696 4898 scope.go:122] "RemoveContainer" containerID="3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924" Nov 03 22:51:58.432060 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.431985 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9" Nov 03 22:51:58.481774 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.481662 4898 scope.go:122] "RemoveContainer" containerID="3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924" Nov 03 22:51:58.482089 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:51:58.482026 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924\": container with ID starting with 3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924 not found: ID does not exist" containerID="3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924" Nov 03 22:51:58.482089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.482060 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924"} err="failed to get container status \"3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924\": rpc error: code = NotFound desc = could not find container \"3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924\": container with ID starting with 3827c3af8175f34a7bb1b2b03da17b9819f9ba72bb8f147997652712233eb924 not found: ID does not exist" Nov 03 22:51:58.617178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.617139 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9"] Nov 03 22:51:58.665867 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.665835 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-jxgb9"] Nov 03 22:51:58.907629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:51:58.907549 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" path="/var/lib/kubelet/pods/2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20/volumes" Nov 03 22:52:01.054014 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.053983 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f"] Nov 03 22:52:01.054626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.054611 4898 cpu_manager.go:414] "containerMap: removing container" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" Nov 03 22:52:01.054666 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.054627 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" Nov 03 22:52:01.054753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.054744 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eeb37c6-d4ef-46bc-b61f-cef76ff7fd20" containerName="istio-proxy" Nov 03 22:52:01.059833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.059814 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.116198 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.116162 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f"] Nov 03 22:52:01.142528 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.142489 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-metrics-certs\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.142700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.142545 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.142743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.142712 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldn5\" (UniqueName: \"kubernetes.io/projected/a76a7d14-6ee0-4ce0-a090-4249044af855-kube-api-access-rldn5\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.142782 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.142769 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76a7d14-6ee0-4ce0-a090-4249044af855-openshift-service-ca\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.161990 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.161961 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2627c5-serving-cert\"" Nov 03 22:52:01.244598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.244566 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldn5\" (UniqueName: \"kubernetes.io/projected/a76a7d14-6ee0-4ce0-a090-4249044af855-kube-api-access-rldn5\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.244843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.244609 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76a7d14-6ee0-4ce0-a090-4249044af855-openshift-service-ca\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.244843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.244829 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-metrics-certs\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.244978 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.244862 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.245069 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:01.245045 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-serving-cert: secret "maas-api-e2e-api-a-2627c5-serving-cert" not found Nov 03 22:52:01.245148 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:01.245136 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls podName:a76a7d14-6ee0-4ce0-a090-4249044af855 nodeName:}" failed. No retries permitted until 2026-11-03 22:52:01.74511545 +0000 UTC m=+1764.957884490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls") pod "maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" (UID: "a76a7d14-6ee0-4ce0-a090-4249044af855") : secret "maas-api-e2e-api-a-2627c5-serving-cert" not found Nov 03 22:52:01.245380 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.245355 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76a7d14-6ee0-4ce0-a090-4249044af855-openshift-service-ca\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.247219 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.247200 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-metrics-certs\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.312888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.312782 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldn5\" (UniqueName: \"kubernetes.io/projected/a76a7d14-6ee0-4ce0-a090-4249044af855-kube-api-access-rldn5\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.760246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.760207 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.762725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.762701 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls\") pod \"maas-api-e2e-api-a-2627c5-77976c88c4-49d5f\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:01.974105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:01.974073 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:02.357725 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:02.357690 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f"] Nov 03 22:52:02.358859 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:52:02.358777 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76a7d14_6ee0_4ce0_a090_4249044af855.slice/crio-a8ef296531062c4733952b619c6c10751be53a0ce057be8e26fd124c4a7b72fe WatchSource:0}: Error finding container a8ef296531062c4733952b619c6c10751be53a0ce057be8e26fd124c4a7b72fe: Status 404 returned error can't find the container with id a8ef296531062c4733952b619c6c10751be53a0ce057be8e26fd124c4a7b72fe Nov 03 22:52:02.448083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:02.448049 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" event={"ID":"a76a7d14-6ee0-4ce0-a090-4249044af855","Type":"ContainerStarted","Data":"a8ef296531062c4733952b619c6c10751be53a0ce057be8e26fd124c4a7b72fe"} Nov 03 22:52:04.464746 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:04.464707 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" event={"ID":"a76a7d14-6ee0-4ce0-a090-4249044af855","Type":"ContainerStarted","Data":"77fdfde5d2b267754a7ea8f5c93d70bd7fb6662b280c447d8f6e2a965ac6a941"} Nov 03 22:52:04.465187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:04.464828 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:04.584596 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:04.584545 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" podStartSLOduration=2.126463886 podStartE2EDuration="3.584531831s" totalImagesPullingTime="1.458067945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:04.582679287 +0000 UTC m=+1767.795448360" watchObservedRunningTime="2026-11-03 22:52:04.584531831 +0000 UTC m=+1767.797300923" Nov 03 22:52:04.957919 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:04.957889 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7243-d7bafc"] Nov 03 22:52:04.962116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:04.962096 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:05.021810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:05.017942 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-d7bafc"] Nov 03 22:52:05.021810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:05.020684 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j4jqv\"" Nov 03 22:52:05.115662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:05.115632 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9676\" (UniqueName: \"kubernetes.io/projected/6b57b80e-50bc-4ee3-92e0-5bb58721415d-kube-api-access-s9676\") pod \"test-curl-7243-d7bafc\" (UID: \"6b57b80e-50bc-4ee3-92e0-5bb58721415d\") " pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:05.217371 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:05.217289 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9676\" (UniqueName: \"kubernetes.io/projected/6b57b80e-50bc-4ee3-92e0-5bb58721415d-kube-api-access-s9676\") pod \"test-curl-7243-d7bafc\" (UID: \"6b57b80e-50bc-4ee3-92e0-5bb58721415d\") " pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:05.278614 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:05.278578 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9676\" (UniqueName: \"kubernetes.io/projected/6b57b80e-50bc-4ee3-92e0-5bb58721415d-kube-api-access-s9676\") pod \"test-curl-7243-d7bafc\" (UID: \"6b57b80e-50bc-4ee3-92e0-5bb58721415d\") " pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:05.575103 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:05.575070 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:05.952331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:05.952305 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-d7bafc"] Nov 03 22:52:05.954115 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:52:05.954074 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b57b80e_50bc_4ee3_92e0_5bb58721415d.slice/crio-ec95ac10aac7f0d486daecf65fabcd1ed2e83da6e20abe0436a9cf71eb56523c WatchSource:0}: Error finding container ec95ac10aac7f0d486daecf65fabcd1ed2e83da6e20abe0436a9cf71eb56523c: Status 404 returned error can't find the container with id ec95ac10aac7f0d486daecf65fabcd1ed2e83da6e20abe0436a9cf71eb56523c Nov 03 22:52:06.481034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:06.480982 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-d7bafc" event={"ID":"6b57b80e-50bc-4ee3-92e0-5bb58721415d","Type":"ContainerStarted","Data":"6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377"} Nov 03 22:52:06.481034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:06.481040 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-d7bafc" event={"ID":"6b57b80e-50bc-4ee3-92e0-5bb58721415d","Type":"ContainerStarted","Data":"ec95ac10aac7f0d486daecf65fabcd1ed2e83da6e20abe0436a9cf71eb56523c"} Nov 03 22:52:06.603034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:06.601041 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7243-d7bafc" podStartSLOduration=2.601019464 podStartE2EDuration="2.601019464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:06.597428836 +0000 UTC m=+1769.810197934" watchObservedRunningTime="2026-11-03 22:52:06.601019464 +0000 UTC m=+1769.813788577" Nov 03 22:52:08.130279 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:08.130247 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7243-d7bafc"] Nov 03 22:52:08.148722 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:08.148689 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7243-d7bafc"] Nov 03 22:52:08.493894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:08.493804 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7243-d7bafc" podUID="6b57b80e-50bc-4ee3-92e0-5bb58721415d" containerName="test-curl-7243-d7bafc" containerID="cri-o://6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377" gracePeriod=2 Nov 03 22:52:08.545624 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:08.545581 4898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7243-d7bafc\" is forbidden: User \"system:node:ip-10-0-139-173.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-173.us-west-2.compute.internal' and this object" podUID="6b57b80e-50bc-4ee3-92e0-5bb58721415d" pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:10.477809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:10.477759 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:52:10.770647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:10.770621 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:10.897680 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:10.897644 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b57b80e-50bc-4ee3-92e0-5bb58721415d-kube-api-access-s9676\" (UniqueName: \"kubernetes.io/projected/6b57b80e-50bc-4ee3-92e0-5bb58721415d-kube-api-access-s9676\") pod \"6b57b80e-50bc-4ee3-92e0-5bb58721415d\" (UID: \"6b57b80e-50bc-4ee3-92e0-5bb58721415d\") " Nov 03 22:52:10.900479 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:10.900443 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b57b80e-50bc-4ee3-92e0-5bb58721415d-kube-api-access-s9676" pod "6b57b80e-50bc-4ee3-92e0-5bb58721415d" (UID: "6b57b80e-50bc-4ee3-92e0-5bb58721415d"). InnerVolumeSpecName "kube-api-access-s9676". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:10.909266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:10.909231 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b57b80e-50bc-4ee3-92e0-5bb58721415d" path="/var/lib/kubelet/pods/6b57b80e-50bc-4ee3-92e0-5bb58721415d/volumes" Nov 03 22:52:11.000373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:11.000262 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9676\" (UniqueName: \"kubernetes.io/projected/6b57b80e-50bc-4ee3-92e0-5bb58721415d-kube-api-access-s9676\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:11.525590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:11.525543 4898 generic.go:356] "Generic (PLEG): container finished" podID="6b57b80e-50bc-4ee3-92e0-5bb58721415d" containerID="6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377" exitCode=137 Nov 03 22:52:11.526078 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:11.525651 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-d7bafc" Nov 03 22:52:11.526078 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:11.525706 4898 scope.go:122] "RemoveContainer" containerID="6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377" Nov 03 22:52:11.535277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:11.535256 4898 scope.go:122] "RemoveContainer" containerID="6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377" Nov 03 22:52:11.535592 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:11.535573 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377\": container with ID starting with 6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377 not found: ID does not exist" containerID="6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377" Nov 03 22:52:11.535650 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:11.535602 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377"} err="failed to get container status \"6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377\": rpc error: code = NotFound desc = could not find container \"6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377\": container with ID starting with 6735c79856e5622e74406fe99d04c5763e15495957ab20a6112333ffded54377 not found: ID does not exist" Nov 03 22:52:16.605199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.605161 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc"] Nov 03 22:52:16.605804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.605776 4898 cpu_manager.go:414] "containerMap: removing container" podUID="6b57b80e-50bc-4ee3-92e0-5bb58721415d" containerName="test-curl-7243-d7bafc" Nov 03 22:52:16.605854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.605805 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b57b80e-50bc-4ee3-92e0-5bb58721415d" containerName="test-curl-7243-d7bafc" Nov 03 22:52:16.605958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.605948 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b57b80e-50bc-4ee3-92e0-5bb58721415d" containerName="test-curl-7243-d7bafc" Nov 03 22:52:16.611038 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.611017 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.666617 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.666582 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc"] Nov 03 22:52:16.675338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.675308 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.675545 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.675514 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-metrics-certs\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.675643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.675627 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.675687 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.675676 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52q2w\" (UniqueName: \"kubernetes.io/projected/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-kube-api-access-52q2w\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.712668 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.712630 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2420f6-serving-cert\"" Nov 03 22:52:16.776808 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.776733 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-metrics-certs\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.777292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.777267 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.778102 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.778078 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.778217 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.778198 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52q2w\" (UniqueName: \"kubernetes.io/projected/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-kube-api-access-52q2w\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.778399 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.778382 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.778609 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:16.778591 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-serving-cert: secret "maas-api-e2e-inf-a-2420f6-serving-cert" not found Nov 03 22:52:16.778683 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:16.778670 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls podName:c0f3f17a-a76b-47cb-910b-706b8f01bbb1 nodeName:}" failed. No retries permitted until 2026-11-03 22:52:17.278647887 +0000 UTC m=+1780.491416927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls") pod "maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" (UID: "c0f3f17a-a76b-47cb-910b-706b8f01bbb1") : secret "maas-api-e2e-inf-a-2420f6-serving-cert" not found Nov 03 22:52:16.779696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.779675 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-metrics-certs\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:16.877769 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:16.874403 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52q2w\" (UniqueName: \"kubernetes.io/projected/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-kube-api-access-52q2w\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:17.286973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:17.286888 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:17.289357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:17.289330 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:17.525383 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:17.525352 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:17.918059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:17.914922 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc"] Nov 03 22:52:17.920217 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:52:17.920186 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f3f17a_a76b_47cb_910b_706b8f01bbb1.slice/crio-7ef4116f07f5349dbad7c1a288cc1b92728764b37c41136ba58a2483e70db71b WatchSource:0}: Error finding container 7ef4116f07f5349dbad7c1a288cc1b92728764b37c41136ba58a2483e70db71b: Status 404 returned error can't find the container with id 7ef4116f07f5349dbad7c1a288cc1b92728764b37c41136ba58a2483e70db71b Nov 03 22:52:18.578844 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:18.578801 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" event={"ID":"c0f3f17a-a76b-47cb-910b-706b8f01bbb1","Type":"ContainerStarted","Data":"7ef4116f07f5349dbad7c1a288cc1b92728764b37c41136ba58a2483e70db71b"} Nov 03 22:52:19.588168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:19.588134 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" event={"ID":"c0f3f17a-a76b-47cb-910b-706b8f01bbb1","Type":"ContainerStarted","Data":"3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681"} Nov 03 22:52:19.588621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:19.588288 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:19.708757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:19.708691 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" podStartSLOduration=2.720241725 podStartE2EDuration="3.708677252s" totalImagesPullingTime="988.435527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:19.707728388 +0000 UTC m=+1782.920497440" watchObservedRunningTime="2026-11-03 22:52:19.708677252 +0000 UTC m=+1782.921446302" Nov 03 22:52:19.910049 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:19.909955 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:52:25.602041 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:25.602012 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:52:31.004321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.004287 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27"] Nov 03 22:52:31.004818 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.004504 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" podUID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" containerName="discovery" containerID="cri-o://4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da" gracePeriod=30 Nov 03 22:52:31.343981 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.343949 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:52:31.391455 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.391424 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-cacerts\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-cacerts\") pod \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " Nov 03 22:52:31.391636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.391463 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-token\") pod \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " Nov 03 22:52:31.391636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.391525 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcf8def-9505-4d56-a7b2-1f979519ebc0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf8def-9505-4d56-a7b2-1f979519ebc0-local-certs\") pod \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " Nov 03 22:52:31.391636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.391593 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-ca-configmap\") pod \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " Nov 03 22:52:31.391821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.391725 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-kubeconfig\") pod \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " Nov 03 22:52:31.391821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.391802 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-dns-cert\") pod \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " Nov 03 22:52:31.391960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.391940 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-kube-api-access-9rsxb\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-kube-api-access-9rsxb\") pod \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\" (UID: \"bbcf8def-9505-4d56-a7b2-1f979519ebc0\") " Nov 03 22:52:31.392101 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.392072 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-ca-configmap" pod "bbcf8def-9505-4d56-a7b2-1f979519ebc0" (UID: "bbcf8def-9505-4d56-a7b2-1f979519ebc0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:31.392650 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.392625 4898 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-ca-configmap\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:31.399291 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.399259 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-kube-api-access-9rsxb" pod "bbcf8def-9505-4d56-a7b2-1f979519ebc0" (UID: "bbcf8def-9505-4d56-a7b2-1f979519ebc0"). InnerVolumeSpecName "kube-api-access-9rsxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:31.400675 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.400645 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-token" pod "bbcf8def-9505-4d56-a7b2-1f979519ebc0" (UID: "bbcf8def-9505-4d56-a7b2-1f979519ebc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:31.400836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.400761 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-cacerts" pod "bbcf8def-9505-4d56-a7b2-1f979519ebc0" (UID: "bbcf8def-9505-4d56-a7b2-1f979519ebc0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:31.402399 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.402374 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcf8def-9505-4d56-a7b2-1f979519ebc0-local-certs" pod "bbcf8def-9505-4d56-a7b2-1f979519ebc0" (UID: "bbcf8def-9505-4d56-a7b2-1f979519ebc0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:31.402586 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.402551 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-dns-cert" pod "bbcf8def-9505-4d56-a7b2-1f979519ebc0" (UID: "bbcf8def-9505-4d56-a7b2-1f979519ebc0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:31.407253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.407223 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-kubeconfig" pod "bbcf8def-9505-4d56-a7b2-1f979519ebc0" (UID: "bbcf8def-9505-4d56-a7b2-1f979519ebc0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:31.494051 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.494011 4898 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-csr-dns-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:31.494051 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.494049 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rsxb\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-kube-api-access-9rsxb\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:31.494051 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.494061 4898 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-cacerts\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:31.494321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.494070 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:31.494321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.494078 4898 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf8def-9505-4d56-a7b2-1f979519ebc0-local-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:31.494321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.494087 4898 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbcf8def-9505-4d56-a7b2-1f979519ebc0-istio-kubeconfig\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:31.745129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.745084 4898 generic.go:356] "Generic (PLEG): container finished" podID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" containerID="4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da" exitCode=0 Nov 03 22:52:31.745328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.745136 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" Nov 03 22:52:31.745328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.745159 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" event={"ID":"bbcf8def-9505-4d56-a7b2-1f979519ebc0","Type":"ContainerDied","Data":"4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da"} Nov 03 22:52:31.745328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.745218 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27" event={"ID":"bbcf8def-9505-4d56-a7b2-1f979519ebc0","Type":"ContainerDied","Data":"04a4be383e727e73df14b3a65b51964212bacc9787fd49d226fae202f7489387"} Nov 03 22:52:31.745328 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.745245 4898 scope.go:122] "RemoveContainer" containerID="4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da" Nov 03 22:52:31.759099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.759074 4898 scope.go:122] "RemoveContainer" containerID="4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da" Nov 03 22:52:31.759415 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:31.759391 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da\": container with ID starting with 4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da not found: ID does not exist" containerID="4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da" Nov 03 22:52:31.759526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.759422 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da"} err="failed to get container status \"4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da\": rpc error: code = NotFound desc = could not find container \"4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da\": container with ID starting with 4223207a36209e5597b51f65fc82768f0d1ac05535165a05da5fdf64a4d111da not found: ID does not exist" Nov 03 22:52:31.772999 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.772951 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:52:31.773179 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.773040 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:52:31.932611 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.932576 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27"] Nov 03 22:52:31.938538 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:31.938509 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmb27"] Nov 03 22:52:32.756521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:32.756439 4898 generic.go:356] "Generic (PLEG): container finished" podID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerID="922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e" exitCode=0 Nov 03 22:52:32.757054 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:32.756516 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" event={"ID":"08b55968-7e4c-4508-b0b4-a64ad6eeb74d","Type":"ContainerDied","Data":"922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e"} Nov 03 22:52:32.757054 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:32.757044 4898 scope.go:122] "RemoveContainer" containerID="922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e" Nov 03 22:52:32.759347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:32.759315 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:52:32.759429 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:32.759405 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:52:32.759466 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:32.759452 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:52:32.909519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:32.909473 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" path="/var/lib/kubelet/pods/bbcf8def-9505-4d56-a7b2-1f979519ebc0/volumes" Nov 03 22:52:33.804731 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:33.804691 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" event={"ID":"08b55968-7e4c-4508-b0b4-a64ad6eeb74d","Type":"ContainerStarted","Data":"aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec"} Nov 03 22:52:34.108470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:34.108435 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:52:34.108642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:34.108550 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:52:34.108642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:34.108595 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:52:35.108487 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:35.108448 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:52:35.108927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:35.108503 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:52:36.108964 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:36.108925 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:52:36.109438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:36.108990 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:52:36.997365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:36.997320 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz"] Nov 03 22:52:36.998299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:36.998198 4898 cpu_manager.go:414] "containerMap: removing container" podUID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" containerName="discovery" Nov 03 22:52:36.998299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:36.998296 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" containerName="discovery" Nov 03 22:52:36.998507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:36.998482 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcf8def-9505-4d56-a7b2-1f979519ebc0" containerName="discovery" Nov 03 22:52:37.001660 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.001641 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.014718 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.014686 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz"] Nov 03 22:52:37.064133 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.064086 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2769cb\"" Nov 03 22:52:37.109841 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.109142 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:52:37.127399 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.125145 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:52:37.167727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.167679 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvphk\" (UniqueName: \"kubernetes.io/projected/ca719cff-42d6-4515-86fb-c72eca762d1d-kube-api-access-qvphk\") pod \"payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.309130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.270431 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2769cb-dockercfg-qxkn4\"" Nov 03 22:52:37.309741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.309508 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca719cff-42d6-4515-86fb-c72eca762d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.322961 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.321241 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x"] Nov 03 22:52:37.387963 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.373121 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x"] Nov 03 22:52:37.387963 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.373480 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:37.410434 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.410395 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca719cff-42d6-4515-86fb-c72eca762d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.410639 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.410621 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvphk\" (UniqueName: \"kubernetes.io/projected/ca719cff-42d6-4515-86fb-c72eca762d1d-kube-api-access-qvphk\") pod \"payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.411322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.411295 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca719cff-42d6-4515-86fb-c72eca762d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.418040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.418005 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:52:37.424493 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.424471 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:52:37.427797 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.427774 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:52:37.433298 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.433283 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:52:37.472810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.472526 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvphk\" (UniqueName: \"kubernetes.io/projected/ca719cff-42d6-4515-86fb-c72eca762d1d-kube-api-access-qvphk\") pod \"payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.511480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.511449 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495459c3-278c-4bdd-a2cc-6ce3b11023e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:37.511632 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.511571 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzr2\" (UniqueName: \"kubernetes.io/projected/495459c3-278c-4bdd-a2cc-6ce3b11023e8-kube-api-access-gtzr2\") pod \"payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:37.613439 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.613396 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495459c3-278c-4bdd-a2cc-6ce3b11023e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:37.613622 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.613485 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzr2\" (UniqueName: \"kubernetes.io/projected/495459c3-278c-4bdd-a2cc-6ce3b11023e8-kube-api-access-gtzr2\") pod \"payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:37.614164 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.614143 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495459c3-278c-4bdd-a2cc-6ce3b11023e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:37.623394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.623367 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:37.676664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.676624 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzr2\" (UniqueName: \"kubernetes.io/projected/495459c3-278c-4bdd-a2cc-6ce3b11023e8-kube-api-access-gtzr2\") pod \"payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:37.703038 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:37.703004 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:38.076032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.075997 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz"] Nov 03 22:52:38.108877 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.108850 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:52:38.108982 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.108898 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:52:38.173712 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.173578 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x"] Nov 03 22:52:38.179565 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:52:38.179534 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495459c3_278c_4bdd_a2cc_6ce3b11023e8.slice/crio-d348c795dabbf585b73bf59dee468434ec33b61292f4c1255734f9faf7bf3cff WatchSource:0}: Error finding container d348c795dabbf585b73bf59dee468434ec33b61292f4c1255734f9faf7bf3cff: Status 404 returned error can't find the container with id d348c795dabbf585b73bf59dee468434ec33b61292f4c1255734f9faf7bf3cff Nov 03 22:52:38.887713 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.887670 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" event={"ID":"495459c3-278c-4bdd-a2cc-6ce3b11023e8","Type":"ContainerStarted","Data":"edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81"} Nov 03 22:52:38.887984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.887733 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" event={"ID":"495459c3-278c-4bdd-a2cc-6ce3b11023e8","Type":"ContainerStarted","Data":"d348c795dabbf585b73bf59dee468434ec33b61292f4c1255734f9faf7bf3cff"} Nov 03 22:52:38.887984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.887916 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:38.889557 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.889532 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" event={"ID":"ca719cff-42d6-4515-86fb-c72eca762d1d","Type":"ContainerStarted","Data":"798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87"} Nov 03 22:52:38.889680 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.889569 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" event={"ID":"ca719cff-42d6-4515-86fb-c72eca762d1d","Type":"ContainerStarted","Data":"5e0ad725c06eeec21130f9f67259412377bac322ff9a185c690a4b26df29264c"} Nov 03 22:52:38.889680 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:38.889665 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:39.015484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:39.015431 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" podStartSLOduration=2.015416138 podStartE2EDuration="2.015416138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:39.013956866 +0000 UTC m=+1802.226725917" watchObservedRunningTime="2026-11-03 22:52:39.015416138 +0000 UTC m=+1802.228185188" Nov 03 22:52:39.108629 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:39.108596 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:52:39.108823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:39.108652 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:52:39.146262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:39.146145 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" podStartSLOduration=3.146128839 podStartE2EDuration="3.146128839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:39.14420698 +0000 UTC m=+1802.356976050" watchObservedRunningTime="2026-11-03 22:52:39.146128839 +0000 UTC m=+1802.358897892" Nov 03 22:52:40.112780 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:40.112750 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:52:40.113169 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:40.113155 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:52:40.113688 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:40.113669 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:52:48.116823 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.116776 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq"] Nov 03 22:52:48.121572 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.121549 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.170859 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.170817 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d97b63-31a2-40ac-b978-2da735c4eec7-openshift-service-ca\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.171056 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.170920 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xc4m\" (UniqueName: \"kubernetes.io/projected/88d97b63-31a2-40ac-b978-2da735c4eec7-kube-api-access-2xc4m\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.171140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.171051 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.171140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.171086 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-metrics-certs\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.174576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.174540 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-922795-serving-cert\"" Nov 03 22:52:48.178201 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.178176 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq"] Nov 03 22:52:48.280813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.274634 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d97b63-31a2-40ac-b978-2da735c4eec7-openshift-service-ca\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.281013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.280909 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xc4m\" (UniqueName: \"kubernetes.io/projected/88d97b63-31a2-40ac-b978-2da735c4eec7-kube-api-access-2xc4m\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.281084 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.281054 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.281144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.281089 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-metrics-certs\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.283852 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.282511 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d97b63-31a2-40ac-b978-2da735c4eec7-openshift-service-ca\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.283852 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:48.282624 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-922795-serving-cert: secret "maas-api-e2e-api-a-922795-serving-cert" not found Nov 03 22:52:48.283852 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:48.282694 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls podName:88d97b63-31a2-40ac-b978-2da735c4eec7 nodeName:}" failed. No retries permitted until 2026-11-03 22:52:48.782675721 +0000 UTC m=+1811.995444766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls") pod "maas-api-e2e-api-a-922795-566c9947bc-24czq" (UID: "88d97b63-31a2-40ac-b978-2da735c4eec7") : secret "maas-api-e2e-api-a-922795-serving-cert" not found Nov 03 22:52:48.289753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.289726 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-metrics-certs\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.348526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.346598 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xc4m\" (UniqueName: \"kubernetes.io/projected/88d97b63-31a2-40ac-b978-2da735c4eec7-kube-api-access-2xc4m\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.787457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.787420 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:48.790181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:48.790156 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls\") pod \"maas-api-e2e-api-a-922795-566c9947bc-24czq\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:49.040860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:49.040750 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:49.434177 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:49.434150 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq"] Nov 03 22:52:49.435398 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:52:49.435375 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d97b63_31a2_40ac_b978_2da735c4eec7.slice/crio-1c3be47a9c5d416ddf648e5bc26cf5c0c92ccc38092191297dfff5de621b07db WatchSource:0}: Error finding container 1c3be47a9c5d416ddf648e5bc26cf5c0c92ccc38092191297dfff5de621b07db: Status 404 returned error can't find the container with id 1c3be47a9c5d416ddf648e5bc26cf5c0c92ccc38092191297dfff5de621b07db Nov 03 22:52:49.835956 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:49.835921 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:52:49.898865 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:49.898839 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:52:49.899040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:49.898987 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:52:49.965205 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:49.965174 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" event={"ID":"88d97b63-31a2-40ac-b978-2da735c4eec7","Type":"ContainerStarted","Data":"1c3be47a9c5d416ddf648e5bc26cf5c0c92ccc38092191297dfff5de621b07db"} Nov 03 22:52:50.976682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:50.976635 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" event={"ID":"88d97b63-31a2-40ac-b978-2da735c4eec7","Type":"ContainerStarted","Data":"f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa"} Nov 03 22:52:50.977102 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:50.976721 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:51.140079 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:51.140018 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" podStartSLOduration=2.214291865 podStartE2EDuration="3.139997817s" totalImagesPullingTime="925.705952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:51.132332802 +0000 UTC m=+1814.345101863" watchObservedRunningTime="2026-11-03 22:52:51.139997817 +0000 UTC m=+1814.352766866" Nov 03 22:52:52.952878 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:52.952844 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6"] Nov 03 22:52:52.953250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:52.953081 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" containerID="cri-o://a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d" gracePeriod=30 Nov 03 22:52:54.229293 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:54.229252 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9"] Nov 03 22:52:54.229735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:54.229467 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" containerID="cri-o://efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69" gracePeriod=30 Nov 03 22:52:56.283426 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:56.283388 4898 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:56.283848 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:56.283440 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:56.989123 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:56.989090 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:52:57.663840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:57.663770 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:57.664438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:57.664407 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:58.228783 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.228722 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:52:58.322058 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322028 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-socket\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322144 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-data\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322185 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-credential-socket\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322318 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322234 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-certs\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322318 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322270 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-envoy\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322318 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322287 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-socket" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.322318 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322306 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-kube-api-access-xplfm\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-kube-api-access-xplfm\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322486 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322400 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-istio-token\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-istio-token\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322486 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322413 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-data" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.322571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322489 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35089b8e-3361-4bbb-844b-7284485d726f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35089b8e-3361-4bbb-844b-7284485d726f-istio-podinfo\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322510 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-credential-socket" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.322571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322540 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35089b8e-3361-4bbb-844b-7284485d726f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35089b8e-3361-4bbb-844b-7284485d726f-istiod-ca-cert\") pod \"35089b8e-3361-4bbb-844b-7284485d726f\" (UID: \"35089b8e-3361-4bbb-844b-7284485d726f\") " Nov 03 22:52:58.322720 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322635 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-certs" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.323010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.322980 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35089b8e-3361-4bbb-844b-7284485d726f-istiod-ca-cert" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:58.323287 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.323263 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.323407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.323294 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.323407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.323311 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.323407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.323328 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.323407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.323343 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35089b8e-3361-4bbb-844b-7284485d726f-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.324762 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.324734 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-kube-api-access-xplfm" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "kube-api-access-xplfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:58.324762 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.324742 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-envoy" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.324960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.324756 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-istio-token" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:58.324960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.324810 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35089b8e-3361-4bbb-844b-7284485d726f-istio-podinfo" pod "35089b8e-3361-4bbb-844b-7284485d726f" (UID: "35089b8e-3361-4bbb-844b-7284485d726f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:58.424921 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.424889 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35089b8e-3361-4bbb-844b-7284485d726f-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.424921 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.424918 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xplfm\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-kube-api-access-xplfm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.424921 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.424929 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35089b8e-3361-4bbb-844b-7284485d726f-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.425170 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:58.424938 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35089b8e-3361-4bbb-844b-7284485d726f-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.037156 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.037116 4898 generic.go:356] "Generic (PLEG): container finished" podID="35089b8e-3361-4bbb-844b-7284485d726f" containerID="a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d" exitCode=0 Nov 03 22:52:59.037700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.037172 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" Nov 03 22:52:59.037900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.037165 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" event={"ID":"35089b8e-3361-4bbb-844b-7284485d726f","Type":"ContainerDied","Data":"a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d"} Nov 03 22:52:59.037900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.037847 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6" event={"ID":"35089b8e-3361-4bbb-844b-7284485d726f","Type":"ContainerDied","Data":"3cd23ce52ba1d099c758bde5a0431b53576da5ad41afba5ec9d08f7435de1919"} Nov 03 22:52:59.037900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.037884 4898 scope.go:122] "RemoveContainer" containerID="a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d" Nov 03 22:52:59.051246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.051224 4898 scope.go:122] "RemoveContainer" containerID="a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d" Nov 03 22:52:59.051680 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:52:59.051655 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d\": container with ID starting with a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d not found: ID does not exist" containerID="a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d" Nov 03 22:52:59.051779 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.051691 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d"} err="failed to get container status \"a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d\": rpc error: code = NotFound desc = could not find container \"a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d\": container with ID starting with a0ad31928a49bdbf7d2aa8f3fd657ea7f0ddca8c8250916e3bd0ec1e7172ca3d not found: ID does not exist" Nov 03 22:52:59.208700 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.208624 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6"] Nov 03 22:52:59.258955 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.258923 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-gpbv6"] Nov 03 22:52:59.523028 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.523003 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:52:59.654643 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.654611 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-kube-api-access-n8rhr\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-kube-api-access-n8rhr\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.654872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.654670 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istiod-ca-cert\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.654872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.654717 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-token\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-token\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.654872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.654784 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-certs\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.654872 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.654854 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-podinfo\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.655138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.654897 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-credential-socket\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.655138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.654957 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-envoy\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.655138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.655003 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-socket\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.655138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.655045 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-data\") pod \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\" (UID: \"5057cf70-852c-412d-bf26-dbf1f53c4c7b\") " Nov 03 22:52:59.656243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.655808 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istiod-ca-cert" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:59.656243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.656037 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-data" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:59.656243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.656123 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-socket" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:59.656243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.656167 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-credential-socket" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:59.656802 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.656742 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-certs" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:59.659902 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.659835 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-token" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:59.660276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.660252 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-podinfo" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:59.660519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.660489 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-kube-api-access-n8rhr" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "kube-api-access-n8rhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:59.663251 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.663206 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-envoy" pod "5057cf70-852c-412d-bf26-dbf1f53c4c7b" (UID: "5057cf70-852c-412d-bf26-dbf1f53c4c7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:59.757315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757240 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757272 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757283 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757293 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757302 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rhr\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-kube-api-access-n8rhr\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757312 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757321 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757330 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5057cf70-852c-412d-bf26-dbf1f53c4c7b-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.757638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:52:59.757340 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5057cf70-852c-412d-bf26-dbf1f53c4c7b-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:00.050764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.050673 4898 generic.go:356] "Generic (PLEG): container finished" podID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerID="efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69" exitCode=0 Nov 03 22:53:00.050764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.050739 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" event={"ID":"5057cf70-852c-412d-bf26-dbf1f53c4c7b","Type":"ContainerDied","Data":"efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69"} Nov 03 22:53:00.051325 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.050823 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" event={"ID":"5057cf70-852c-412d-bf26-dbf1f53c4c7b","Type":"ContainerDied","Data":"0c49c349b3a51613d730075605c15ffaf083fa04099b6a4612971277aa245bb0"} Nov 03 22:53:00.051325 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.050861 4898 scope.go:122] "RemoveContainer" containerID="efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69" Nov 03 22:53:00.051325 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.051006 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9" Nov 03 22:53:00.068079 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.068053 4898 scope.go:122] "RemoveContainer" containerID="efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69" Nov 03 22:53:00.068397 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:00.068376 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69\": container with ID starting with efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69 not found: ID does not exist" containerID="efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69" Nov 03 22:53:00.068495 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.068404 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69"} err="failed to get container status \"efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69\": rpc error: code = NotFound desc = could not find container \"efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69\": container with ID starting with efcff30cdf72b3f6dc869b9d84f1d9e77e46c0d524781af9930a346615c32c69 not found: ID does not exist" Nov 03 22:53:00.223707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.223664 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9"] Nov 03 22:53:00.274070 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.274035 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-fwck9"] Nov 03 22:53:00.908893 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.908857 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35089b8e-3361-4bbb-844b-7284485d726f" path="/var/lib/kubelet/pods/35089b8e-3361-4bbb-844b-7284485d726f/volumes" Nov 03 22:53:00.909353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:00.909339 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" path="/var/lib/kubelet/pods/5057cf70-852c-412d-bf26-dbf1f53c4c7b/volumes" Nov 03 22:53:02.993238 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.993199 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4"] Nov 03 22:53:02.994273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.994243 4898 cpu_manager.go:414] "containerMap: removing container" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" Nov 03 22:53:02.994273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.994273 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" Nov 03 22:53:02.994502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.994329 4898 cpu_manager.go:414] "containerMap: removing container" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" Nov 03 22:53:02.994502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.994337 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" Nov 03 22:53:02.994502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.994478 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="35089b8e-3361-4bbb-844b-7284485d726f" containerName="istio-proxy" Nov 03 22:53:02.994502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.994496 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5057cf70-852c-412d-bf26-dbf1f53c4c7b" containerName="istio-proxy" Nov 03 22:53:02.999987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:02.999961 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.062942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.060270 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4"] Nov 03 22:53:03.098399 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.098351 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-65913e-serving-cert\"" Nov 03 22:53:03.102194 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.102160 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.102347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.102239 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69wd2\" (UniqueName: \"kubernetes.io/projected/1294603d-a69d-47a6-9a02-42051d7afefe-kube-api-access-69wd2\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.102527 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.102493 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1294603d-a69d-47a6-9a02-42051d7afefe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.102573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.102559 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.204070 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.204024 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1294603d-a69d-47a6-9a02-42051d7afefe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.204275 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.204100 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.204275 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.204197 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.204275 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:03.204259 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-serving-cert: secret "maas-api-e2e-shared-b-w2-65913e-serving-cert" not found Nov 03 22:53:03.204456 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.204270 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wd2\" (UniqueName: \"kubernetes.io/projected/1294603d-a69d-47a6-9a02-42051d7afefe-kube-api-access-69wd2\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.204456 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:03.204373 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls podName:1294603d-a69d-47a6-9a02-42051d7afefe nodeName:}" failed. No retries permitted until 2026-11-03 22:53:03.704350025 +0000 UTC m=+1826.917119075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls") pod "maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" (UID: "1294603d-a69d-47a6-9a02-42051d7afefe") : secret "maas-api-e2e-shared-b-w2-65913e-serving-cert" not found Nov 03 22:53:03.204830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.204810 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1294603d-a69d-47a6-9a02-42051d7afefe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.206863 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.206829 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.270221 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.270129 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69wd2\" (UniqueName: \"kubernetes.io/projected/1294603d-a69d-47a6-9a02-42051d7afefe-kube-api-access-69wd2\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.712642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.712602 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.715128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.715100 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:03.914272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:03.914238 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:04.306134 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:53:04.306104 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1294603d_a69d_47a6_9a02_42051d7afefe.slice/crio-5820bd520ac8b25fc237f4205f95a9c11884fb61bebffbe274f99952e3ad9202 WatchSource:0}: Error finding container 5820bd520ac8b25fc237f4205f95a9c11884fb61bebffbe274f99952e3ad9202: Status 404 returned error can't find the container with id 5820bd520ac8b25fc237f4205f95a9c11884fb61bebffbe274f99952e3ad9202 Nov 03 22:53:04.306134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:04.306109 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4"] Nov 03 22:53:05.104991 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:05.104946 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" event={"ID":"1294603d-a69d-47a6-9a02-42051d7afefe","Type":"ContainerStarted","Data":"5820bd520ac8b25fc237f4205f95a9c11884fb61bebffbe274f99952e3ad9202"} Nov 03 22:53:06.119442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:06.119399 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" event={"ID":"1294603d-a69d-47a6-9a02-42051d7afefe","Type":"ContainerStarted","Data":"3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4"} Nov 03 22:53:06.119948 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:06.119480 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:06.237269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:06.237218 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" podStartSLOduration=3.291616714 podStartE2EDuration="4.237203206s" totalImagesPullingTime="945.586492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:06.237119153 +0000 UTC m=+1829.449888216" watchObservedRunningTime="2026-11-03 22:53:06.237203206 +0000 UTC m=+1829.449972261" Nov 03 22:53:12.132189 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:12.132157 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:53:19.840831 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:19.840767 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:53:21.309889 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:21.309803 4898 generic.go:356] "Generic (PLEG): container finished" podID="b996d093-0a87-4e45-afc4-4488b00f78da" containerID="01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08" exitCode=0 Nov 03 22:53:21.309889 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:21.309859 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" event={"ID":"b996d093-0a87-4e45-afc4-4488b00f78da","Type":"ContainerDied","Data":"01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08"} Nov 03 22:53:21.310390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:21.310279 4898 scope.go:122] "RemoveContainer" containerID="01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08" Nov 03 22:53:21.311963 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:21.311934 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:21.312084 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:21.312017 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:21.312084 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:21.312064 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:22.318289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.318249 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" event={"ID":"b996d093-0a87-4e45-afc4-4488b00f78da","Type":"ContainerStarted","Data":"551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70"} Nov 03 22:53:22.506569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.506530 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg"] Nov 03 22:53:22.510885 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.510859 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.564384 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.564355 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-0189ef-serving-cert\"" Nov 03 22:53:22.565111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.565075 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99nlv\" (UniqueName: \"kubernetes.io/projected/8c88f565-1f64-4542-9a7a-b2390803b9d7-kube-api-access-99nlv\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.569335 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.569254 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.569982 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.569946 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.570097 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.569999 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c88f565-1f64-4542-9a7a-b2390803b9d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.573846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.573820 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg"] Nov 03 22:53:22.581252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.581110 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:53:22.582285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.582259 4898 patch_prober.go:29] interesting pod/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:53:22.582392 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.582335 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:53:22.682899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.674118 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99nlv\" (UniqueName: \"kubernetes.io/projected/8c88f565-1f64-4542-9a7a-b2390803b9d7-kube-api-access-99nlv\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.682899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.674310 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.682899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.674403 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.682899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.674437 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c88f565-1f64-4542-9a7a-b2390803b9d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.682899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.675239 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c88f565-1f64-4542-9a7a-b2390803b9d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.682899 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:22.676318 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-serving-cert: secret "maas-api-e2e-shared-b-w6-0189ef-serving-cert" not found Nov 03 22:53:22.682899 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:22.676380 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls podName:8c88f565-1f64-4542-9a7a-b2390803b9d7 nodeName:}" failed. No retries permitted until 2026-11-03 22:53:23.176361026 +0000 UTC m=+1846.389130070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls") pod "maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" (UID: "8c88f565-1f64-4542-9a7a-b2390803b9d7") : secret "maas-api-e2e-shared-b-w6-0189ef-serving-cert" not found Nov 03 22:53:22.694267 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.694235 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:22.796708 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:22.796672 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99nlv\" (UniqueName: \"kubernetes.io/projected/8c88f565-1f64-4542-9a7a-b2390803b9d7-kube-api-access-99nlv\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:23.198574 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:23.198530 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:23.201142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:23.201121 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:23.424419 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:23.424378 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:23.581178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:23.581141 4898 patch_prober.go:29] interesting pod/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:53:23.581374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:23.581191 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:53:23.850966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:23.849781 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg"] Nov 03 22:53:24.355813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:24.355303 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" event={"ID":"8c88f565-1f64-4542-9a7a-b2390803b9d7","Type":"ContainerStarted","Data":"86796d8af242d9e0d301cd805d85fc8b2e77324a80ccc45b32befcc999fbb422"} Nov 03 22:53:24.581491 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:24.581450 4898 patch_prober.go:29] interesting pod/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:53:24.581995 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:24.581514 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:53:24.916976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:24.916945 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:53:25.376591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:25.376550 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" event={"ID":"8c88f565-1f64-4542-9a7a-b2390803b9d7","Type":"ContainerStarted","Data":"e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c"} Nov 03 22:53:25.376912 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:25.376889 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:25.495905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:25.495836 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" podStartSLOduration=2.629432665 podStartE2EDuration="3.495822746s" totalImagesPullingTime="866.390081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:25.495055886 +0000 UTC m=+1848.707824953" watchObservedRunningTime="2026-11-03 22:53:25.495822746 +0000 UTC m=+1848.708591797" Nov 03 22:53:25.581299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:25.581264 4898 patch_prober.go:29] interesting pod/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:53:25.581492 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:25.581325 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:53:26.581407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:26.581368 4898 patch_prober.go:29] interesting pod/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:53:26.581873 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:26.581437 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:53:27.581804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:27.581757 4898 patch_prober.go:29] interesting pod/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:53:27.582290 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:27.581842 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:53:28.585900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:28.585869 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:53:28.586882 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:28.586861 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:53:31.390000 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:31.389969 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:53:42.979689 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:42.979649 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x"] Nov 03 22:53:42.986331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:42.986298 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018229 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-credential-socket\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018456 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018334 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-token\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018456 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018435 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2267cf-8669-41cd-acbd-8f02e561853e-istiod-ca-cert\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018584 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018492 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-envoy\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018641 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018614 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-data\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018642 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv272\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-kube-api-access-zv272\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018751 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018718 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-podinfo\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018754 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-socket\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.018846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.018779 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-certs\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.039366 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.039339 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d6296b-openshift-default-dockercfg-mmh6s\"" Nov 03 22:53:43.131809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.131676 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-podinfo\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.131809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.131749 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-socket\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.131809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.131776 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-certs\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.143809 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.132227 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-certs\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.157824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.157257 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-credential-socket\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.157824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.157350 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-token\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.157824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.157431 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2267cf-8669-41cd-acbd-8f02e561853e-istiod-ca-cert\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.157824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.157506 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-envoy\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.157824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.157582 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-data\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.157824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.157612 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv272\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-kube-api-access-zv272\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.195037 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.187218 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-socket\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.197988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.191600 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2267cf-8669-41cd-acbd-8f02e561853e-istiod-ca-cert\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.198188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.191881 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-credential-socket\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.198656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.198627 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-data\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.265523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.265460 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv272\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-kube-api-access-zv272\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.268175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.268147 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-token\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.268381 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.268166 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x"] Nov 03 22:53:43.283818 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.283329 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-envoy\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.294558 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.294524 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-podinfo\") pod \"e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.361569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.361534 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:43.813414 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.813379 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x"] Nov 03 22:53:43.849546 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:53:43.849508 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2267cf_8669_41cd_acbd_8f02e561853e.slice/crio-fae6dcca8ce2f08979ffd3fdc9faf1b088e4999e4229e8c851e57752f7401703 WatchSource:0}: Error finding container fae6dcca8ce2f08979ffd3fdc9faf1b088e4999e4229e8c851e57752f7401703: Status 404 returned error can't find the container with id fae6dcca8ce2f08979ffd3fdc9faf1b088e4999e4229e8c851e57752f7401703 Nov 03 22:53:43.852701 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.852661 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:43.852833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.852750 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:43.852833 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:43.852821 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:44.518988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:44.518944 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" event={"ID":"4d2267cf-8669-41cd-acbd-8f02e561853e","Type":"ContainerStarted","Data":"9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04"} Nov 03 22:53:44.519476 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:44.519009 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" event={"ID":"4d2267cf-8669-41cd-acbd-8f02e561853e","Type":"ContainerStarted","Data":"fae6dcca8ce2f08979ffd3fdc9faf1b088e4999e4229e8c851e57752f7401703"} Nov 03 22:53:44.641036 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:44.640968 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podStartSLOduration=2.640946089 podStartE2EDuration="2.640946089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:44.639560778 +0000 UTC m=+1867.852329822" watchObservedRunningTime="2026-11-03 22:53:44.640946089 +0000 UTC m=+1867.853715141" Nov 03 22:53:45.363175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:45.363142 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:45.369142 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:45.369112 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:45.526323 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:45.526288 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:45.527199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:45.527173 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:53:49.836183 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:49.836109 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:53:56.967691 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:56.967653 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx"] Nov 03 22:53:56.973677 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:56.973654 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.029556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.029517 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9674f5-openshift-default-dockercfg-qjjw4\"" Nov 03 22:53:57.093713 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.093677 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx"] Nov 03 22:53:57.138608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.138563 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-data\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.139073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.139032 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-podinfo\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.139330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.139309 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-socket\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.139535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.139503 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-credential-socket\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.139716 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.139698 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-certs\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.139987 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.139956 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56r5\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-kube-api-access-z56r5\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.140189 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.140169 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-envoy\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.140410 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.140391 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istiod-ca-cert\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.140709 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.140689 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-token\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.268765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.268650 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istiod-ca-cert\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.292159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.292121 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-token\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.292518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.292496 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-data\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.292749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.292732 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-podinfo\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.292894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.292878 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-socket\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.293006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.292992 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-credential-socket\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.293155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.293138 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-certs\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.293348 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.293334 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56r5\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-kube-api-access-z56r5\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.293462 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.293443 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-envoy\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.294679 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.294648 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istiod-ca-cert\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.321025 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.295329 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-socket\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.321261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.295898 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-data\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.321404 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.315271 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-certs\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.321500 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.315527 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-credential-socket\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.332860 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.324116 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-envoy\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.356140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.351459 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-podinfo\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.373088 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.373018 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-token\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.384832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.381226 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56r5\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-kube-api-access-z56r5\") pod \"e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:57.589478 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:57.589441 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:58.103207 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:58.102973 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx"] Nov 03 22:53:58.105034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:58.104998 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:58.105169 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:58.105081 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:58.105169 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:58.105127 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:53:58.685812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:58.684520 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" event={"ID":"661c0a78-c7ec-4875-a439-fafe63b9c2d2","Type":"ContainerStarted","Data":"958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50"} Nov 03 22:53:58.685812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:58.684630 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" event={"ID":"661c0a78-c7ec-4875-a439-fafe63b9c2d2","Type":"ContainerStarted","Data":"1d23ad4197f961b39e381691f28976678868f1b6341e3a6e26a48f47ced7c06d"} Nov 03 22:53:58.812883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:58.810896 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" podStartSLOduration=2.810875022 podStartE2EDuration="2.810875022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:58.806254887 +0000 UTC m=+1882.019023962" watchObservedRunningTime="2026-11-03 22:53:58.810875022 +0000 UTC m=+1882.023644076" Nov 03 22:53:59.114802 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.108907 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8"] Nov 03 22:53:59.114802 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.114800 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.128290 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.128222 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-metrics-certs\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.128421 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.128340 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ed0f01-5467-4f3c-b635-26f79ef2bd59-openshift-service-ca\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.128551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.128501 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rq5k\" (UniqueName: \"kubernetes.io/projected/42ed0f01-5467-4f3c-b635-26f79ef2bd59-kube-api-access-2rq5k\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.128608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.128569 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.158153 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.158116 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8"] Nov 03 22:53:59.180212 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.180183 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2769cb-serving-cert\"" Nov 03 22:53:59.230812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.230650 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rq5k\" (UniqueName: \"kubernetes.io/projected/42ed0f01-5467-4f3c-b635-26f79ef2bd59-kube-api-access-2rq5k\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.231124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.231098 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.231568 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:59.231209 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-serving-cert: secret "maas-api-e2e-api-b-2769cb-serving-cert" not found Nov 03 22:53:59.231568 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:53:59.231390 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls podName:42ed0f01-5467-4f3c-b635-26f79ef2bd59 nodeName:}" failed. No retries permitted until 2026-11-03 22:53:59.731367949 +0000 UTC m=+1882.944136998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls") pod "maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" (UID: "42ed0f01-5467-4f3c-b635-26f79ef2bd59") : secret "maas-api-e2e-api-b-2769cb-serving-cert" not found Nov 03 22:53:59.231738 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.231723 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-metrics-certs\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.232085 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.232063 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ed0f01-5467-4f3c-b635-26f79ef2bd59-openshift-service-ca\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.233078 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.233056 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ed0f01-5467-4f3c-b635-26f79ef2bd59-openshift-service-ca\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.324626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.324573 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-metrics-certs\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.350826 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.332709 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rq5k\" (UniqueName: \"kubernetes.io/projected/42ed0f01-5467-4f3c-b635-26f79ef2bd59-kube-api-access-2rq5k\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.609813 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.604377 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:59.617616 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.617583 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:59.700110 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.700075 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:59.701288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.701263 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:53:59.810161 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.810124 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:53:59.812623 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:53:59.812595 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls\") pod \"maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:54:00.054863 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:00.042382 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:54:00.625060 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:00.625020 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8"] Nov 03 22:54:00.714916 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:00.714863 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" event={"ID":"42ed0f01-5467-4f3c-b635-26f79ef2bd59","Type":"ContainerStarted","Data":"417a3d805d33814da6244ba2174c7fff83b4a6c0d257fa845358717f137ab831"} Nov 03 22:54:02.731714 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:02.731624 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" event={"ID":"42ed0f01-5467-4f3c-b635-26f79ef2bd59","Type":"ContainerStarted","Data":"f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb"} Nov 03 22:54:02.732132 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:02.731729 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:54:02.911576 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:02.911506 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" podStartSLOduration=2.8760132929999997 podStartE2EDuration="3.911488283s" totalImagesPullingTime="1.03547499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:02.910161352 +0000 UTC m=+1886.122930406" watchObservedRunningTime="2026-11-03 22:54:02.911488283 +0000 UTC m=+1886.124257345" Nov 03 22:54:06.281144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.278885 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:54:06.281144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.278969 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:54:06.376967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.376928 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:54:06.377105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.376994 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:54:06.763547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.763517 4898 generic.go:356] "Generic (PLEG): container finished" podID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerID="b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd" exitCode=0 Nov 03 22:54:06.763716 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.763599 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" event={"ID":"952a43e1-5149-45e3-8c2e-55eec1b64d63","Type":"ContainerDied","Data":"b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd"} Nov 03 22:54:06.764068 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.764049 4898 scope.go:122] "RemoveContainer" containerID="b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd" Nov 03 22:54:06.765768 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.765733 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:06.765890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.765835 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:06.765890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:06.765872 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:07.773249 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.773216 4898 generic.go:356] "Generic (PLEG): container finished" podID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerID="81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274" exitCode=0 Nov 03 22:54:07.773753 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.773293 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" event={"ID":"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3","Type":"ContainerDied","Data":"81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274"} Nov 03 22:54:07.773890 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.773872 4898 scope.go:122] "RemoveContainer" containerID="81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274" Nov 03 22:54:07.775551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.775533 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" event={"ID":"952a43e1-5149-45e3-8c2e-55eec1b64d63","Type":"ContainerStarted","Data":"7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4"} Nov 03 22:54:07.775830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.775801 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:07.775915 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.775874 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:07.775957 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.775914 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:08.003606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:07.999994 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:54:08.003606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:08.000109 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:54:08.003606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:08.000152 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 22:54:09.019838 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:09.013146 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:54:09.019838 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:09.013207 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 22:54:09.065979 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:09.065942 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 22:54:09.074364 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:09.074326 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" event={"ID":"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3","Type":"ContainerStarted","Data":"ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c"} Nov 03 22:54:09.285871 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:09.281182 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:54:09.307686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:09.299479 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:54:09.307686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:09.299538 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:54:10.000272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:10.000239 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:54:10.000501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:10.000479 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 22:54:10.282292 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:10.282200 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:54:10.282953 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:10.282919 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:54:11.000535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:11.000487 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:54:11.000744 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:11.000541 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 22:54:11.284613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:11.284515 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:54:11.285284 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:11.285245 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:54:12.001283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:12.001090 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:54:12.001283 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:12.001162 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 22:54:12.279243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:12.279149 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:54:12.279547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:12.279513 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:54:13.000193 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:13.000153 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:54:13.000732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:13.000703 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 22:54:13.276997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:13.276181 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:54:13.276997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:13.276253 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:54:14.004545 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:14.004513 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:54:14.004970 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:14.004819 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:54:14.005507 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:14.005490 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:54:14.275765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:14.275661 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:54:14.275765 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:14.275715 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:54:15.280032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:15.279998 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:54:15.280447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:15.280384 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:54:15.281133 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:15.281114 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:54:19.837530 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:54:19.837482 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:54:24.880221 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:24.880182 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629"] Nov 03 22:54:24.880626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:24.880431 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" containerID="cri-o://5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8" gracePeriod=30 Nov 03 22:54:27.774740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:27.774702 4898 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:54:27.775192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:27.774774 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:54:30.171086 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.171053 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:54:30.249201 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.249159 4898 generic.go:356] "Generic (PLEG): container finished" podID="636b01e1-8ade-44d2-af06-5c85804832e7" containerID="5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8" exitCode=0 Nov 03 22:54:30.249395 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.249217 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" Nov 03 22:54:30.249395 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.249217 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" event={"ID":"636b01e1-8ade-44d2-af06-5c85804832e7","Type":"ContainerDied","Data":"5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8"} Nov 03 22:54:30.249395 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.249351 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629" event={"ID":"636b01e1-8ade-44d2-af06-5c85804832e7","Type":"ContainerDied","Data":"fe4a1c5504cb000659e2794f4e999bc0017be5a8392852930837a59b5538a0c5"} Nov 03 22:54:30.249395 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.249380 4898 scope.go:122] "RemoveContainer" containerID="5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8" Nov 03 22:54:30.261864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.261770 4898 scope.go:122] "RemoveContainer" containerID="5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8" Nov 03 22:54:30.262101 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:54:30.262081 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8\": container with ID starting with 5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8 not found: ID does not exist" containerID="5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8" Nov 03 22:54:30.262172 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.262113 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8"} err="failed to get container status \"5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8\": rpc error: code = NotFound desc = could not find container \"5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8\": container with ID starting with 5f7563cf698b166936d6328f34bbc2dcc370eba4d6b5a466b170b20a953e5ed8 not found: ID does not exist" Nov 03 22:54:30.290480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290409 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-data\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290462 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/636b01e1-8ade-44d2-af06-5c85804832e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636b01e1-8ade-44d2-af06-5c85804832e7-istio-podinfo\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290518 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-certs\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290573 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-socket\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290603 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-kube-api-access-24fpl\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-kube-api-access-24fpl\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290648 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-envoy\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290685 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-istio-token\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-istio-token\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290719 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636b01e1-8ade-44d2-af06-5c85804832e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636b01e1-8ade-44d2-af06-5c85804832e7-istiod-ca-cert\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.290807 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290738 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-data" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:30.291083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.290943 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-credential-socket\") pod \"636b01e1-8ade-44d2-af06-5c85804832e7\" (UID: \"636b01e1-8ade-44d2-af06-5c85804832e7\") " Nov 03 22:54:30.291145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.291089 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-certs" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:30.291476 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.291453 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.291476 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.291473 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.291908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.291578 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-socket" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:30.291908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.291678 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-credential-socket" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:30.291908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.291777 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636b01e1-8ade-44d2-af06-5c85804832e7-istiod-ca-cert" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:30.293171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.293147 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-kube-api-access-24fpl" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "kube-api-access-24fpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:30.293171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.293148 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/636b01e1-8ade-44d2-af06-5c85804832e7-istio-podinfo" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:30.293349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.293335 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-istio-token" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:30.293759 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.293733 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-envoy" pod "636b01e1-8ade-44d2-af06-5c85804832e7" (UID: "636b01e1-8ade-44d2-af06-5c85804832e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:30.393134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.393098 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.393134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.393125 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636b01e1-8ade-44d2-af06-5c85804832e7-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.393134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.393136 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.393134 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.393145 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fpl\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-kube-api-access-24fpl\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.393498 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.393155 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636b01e1-8ade-44d2-af06-5c85804832e7-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.393498 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.393163 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636b01e1-8ade-44d2-af06-5c85804832e7-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.393498 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.393174 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636b01e1-8ade-44d2-af06-5c85804832e7-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.679321 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.679218 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629"] Nov 03 22:54:30.684143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.684109 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-sq629"] Nov 03 22:54:30.910081 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:30.910047 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" path="/var/lib/kubelet/pods/636b01e1-8ade-44d2-af06-5c85804832e7/volumes" Nov 03 22:54:39.167968 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:39.167926 4898 patch_prober.go:29] interesting pod/e2e-api-a-922795-openshift-default-59b46784df-6bc47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:54:39.168428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:39.168014 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:54:42.369406 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:42.369363 4898 generic.go:356] "Generic (PLEG): container finished" podID="ba948794-2afd-408b-bd73-b7f1d520f746" containerID="014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018" exitCode=0 Nov 03 22:54:42.369997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:42.369453 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" event={"ID":"ba948794-2afd-408b-bd73-b7f1d520f746","Type":"ContainerDied","Data":"014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018"} Nov 03 22:54:42.369997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:42.369971 4898 scope.go:122] "RemoveContainer" containerID="014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018" Nov 03 22:54:42.372065 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:42.372033 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:42.372190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:42.372104 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:42.372190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:42.372141 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:54:43.379358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:43.379319 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" event={"ID":"ba948794-2afd-408b-bd73-b7f1d520f746","Type":"ContainerStarted","Data":"819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27"} Nov 03 22:54:43.656897 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:43.656811 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:54:43.657288 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:43.657253 4898 patch_prober.go:29] interesting pod/e2e-api-a-922795-openshift-default-59b46784df-6bc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:54:43.657459 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:43.657437 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:54:44.658206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:44.658165 4898 patch_prober.go:29] interesting pod/e2e-api-a-922795-openshift-default-59b46784df-6bc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:54:44.658978 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:44.658942 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:54:45.655221 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:45.655178 4898 patch_prober.go:29] interesting pod/e2e-api-a-922795-openshift-default-59b46784df-6bc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:54:45.655548 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:45.655514 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:54:46.659204 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:46.659165 4898 patch_prober.go:29] interesting pod/e2e-api-a-922795-openshift-default-59b46784df-6bc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:54:46.659747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:46.659221 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:54:47.660295 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:47.660248 4898 patch_prober.go:29] interesting pod/e2e-api-a-922795-openshift-default-59b46784df-6bc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:54:47.660717 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:47.660321 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:54:48.656063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:48.656022 4898 patch_prober.go:29] interesting pod/e2e-api-a-922795-openshift-default-59b46784df-6bc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:54:48.656373 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:48.656337 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:54:49.659588 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:49.659559 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:54:49.660054 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:49.660035 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:54:49.660672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:49.660653 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:54:49.835442 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:54:49.835409 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:54:54.056249 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:54.056216 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt"] Nov 03 22:54:54.056691 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:54.056436 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" containerID="cri-o://bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45" gracePeriod=30 Nov 03 22:54:59.361399 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.361371 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:54:59.455567 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455527 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-credential-socket\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.455732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455623 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-data\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.455732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455700 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-kube-api-access-xzwvp\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-kube-api-access-xzwvp\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.455845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455744 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-socket\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.455922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455868 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-envoy\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.455980 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455942 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-certs\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.455980 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455862 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-credential-socket" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:59.456091 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.455957 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-data" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:59.456091 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456013 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-podinfo\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.456091 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456056 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df6004b-a782-479f-9966-9c38ad4ed0a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df6004b-a782-479f-9966-9c38ad4ed0a7-istiod-ca-cert\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.456265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456080 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-socket" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:59.456265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456131 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-token\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-token\") pod \"7df6004b-a782-479f-9966-9c38ad4ed0a7\" (UID: \"7df6004b-a782-479f-9966-9c38ad4ed0a7\") " Nov 03 22:54:59.456265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456193 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-certs" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:59.456688 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456489 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df6004b-a782-479f-9966-9c38ad4ed0a7-istiod-ca-cert" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:59.456825 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456810 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.456903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456835 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.456903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456851 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.456903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456864 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df6004b-a782-479f-9966-9c38ad4ed0a7-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.456903 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.456878 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.458536 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.458505 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-envoy" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:59.458691 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.458668 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-kube-api-access-xzwvp" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "kube-api-access-xzwvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:59.458766 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.458702 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-token" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:59.458851 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.458820 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-podinfo" pod "7df6004b-a782-479f-9966-9c38ad4ed0a7" (UID: "7df6004b-a782-479f-9966-9c38ad4ed0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:59.559988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.559807 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwvp\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-kube-api-access-xzwvp\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.559988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.559846 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.559988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.559862 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.559988 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.559875 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df6004b-a782-479f-9966-9c38ad4ed0a7-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:59.587973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.577861 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" Nov 03 22:54:59.587973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.577927 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" event={"ID":"7df6004b-a782-479f-9966-9c38ad4ed0a7","Type":"ContainerDied","Data":"bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45"} Nov 03 22:54:59.587973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.578003 4898 scope.go:122] "RemoveContainer" containerID="bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45" Nov 03 22:54:59.587973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.577743 4898 generic.go:356] "Generic (PLEG): container finished" podID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerID="bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45" exitCode=0 Nov 03 22:54:59.587973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.582997 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt" event={"ID":"7df6004b-a782-479f-9966-9c38ad4ed0a7","Type":"ContainerDied","Data":"9e780cb841661f946e45f43f24a5087ca18f8543c980be0d420c3fcbf1568c9e"} Nov 03 22:54:59.592870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.592839 4898 scope.go:122] "RemoveContainer" containerID="bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45" Nov 03 22:54:59.593526 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:54:59.593457 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45\": container with ID starting with bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45 not found: ID does not exist" containerID="bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45" Nov 03 22:54:59.593526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.593496 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45"} err="failed to get container status \"bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45\": rpc error: code = NotFound desc = could not find container \"bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45\": container with ID starting with bc7f0fa4ace7adafd6d71f50a68f54145e1234104937b9373153affc1df04e45 not found: ID does not exist" Nov 03 22:54:59.749240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.749205 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt"] Nov 03 22:54:59.754365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:54:59.754340 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-n7vdt"] Nov 03 22:55:00.908083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:00.908049 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" path="/var/lib/kubelet/pods/7df6004b-a782-479f-9966-9c38ad4ed0a7/volumes" Nov 03 22:55:11.645477 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.645440 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45"] Nov 03 22:55:11.645918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.645729 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" containerID="cri-o://296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02" gracePeriod=30 Nov 03 22:55:11.693919 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.693883 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw"] Nov 03 22:55:11.694155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.694118 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" containerID="cri-o://66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849" gracePeriod=30 Nov 03 22:55:11.697599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.697571 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt"] Nov 03 22:55:11.697819 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.697761 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" containerID="cri-o://1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86" gracePeriod=30 Nov 03 22:55:11.744376 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.744338 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm"] Nov 03 22:55:11.744562 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:11.744530 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" containerID="cri-o://522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403" gracePeriod=30 Nov 03 22:55:17.013908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.013876 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:55:17.092740 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.092716 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:55:17.140177 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140132 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-envoy\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.140388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140211 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-podinfo\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.140388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140247 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-certs\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140286 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-kube-api-access-8vvzg\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-kube-api-access-8vvzg\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140324 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-kube-api-access-tdxk8\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-kube-api-access-tdxk8\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.140388 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140361 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69134f35-56c1-4e30-95e2-8989973e6a95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69134f35-56c1-4e30-95e2-8989973e6a95-istio-podinfo\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140634 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140407 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-data\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.140634 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140529 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-credential-socket\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.140634 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140574 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-data\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140644 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-envoy\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140775 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140745 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-socket\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.140871 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140777 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-socket\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140871 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140825 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-token\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-token\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.140871 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140865 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69134f35-56c1-4e30-95e2-8989973e6a95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69134f35-56c1-4e30-95e2-8989973e6a95-istiod-ca-cert\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140892 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-credential-socket\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.140967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.140943 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-certs\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.141049 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.141024 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-istio-token\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-istio-token\") pod \"69134f35-56c1-4e30-95e2-8989973e6a95\" (UID: \"69134f35-56c1-4e30-95e2-8989973e6a95\") " Nov 03 22:55:17.141080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.141050 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istiod-ca-cert\") pod \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\" (UID: \"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27\") " Nov 03 22:55:17.142900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.142861 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istiod-ca-cert" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.144012 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-socket" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.144185 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-certs" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.144230 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-socket" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.144329 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69134f35-56c1-4e30-95e2-8989973e6a95-istiod-ca-cert" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.144429 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-credential-socket" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.144620 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-data" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.146680 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-data" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.146745 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-certs" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.147128 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.146879 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-credential-socket" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.149181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.149146 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-istio-token" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.155067 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.155035 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-podinfo" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:17.155257 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.155084 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-kube-api-access-8vvzg" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "kube-api-access-8vvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.155257 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.155155 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-envoy" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.156758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.156685 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-kube-api-access-tdxk8" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "kube-api-access-tdxk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.157040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.156915 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-envoy" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.158019 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.157935 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-token" pod "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" (UID: "aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.160289 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.160266 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69134f35-56c1-4e30-95e2-8989973e6a95-istio-podinfo" pod "69134f35-56c1-4e30-95e2-8989973e6a95" (UID: "69134f35-56c1-4e30-95e2-8989973e6a95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:17.166655 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.166633 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:55:17.170336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.170318 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:55:17.242309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242219 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-socket\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.242309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242295 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-credential-socket\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242354 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-envoy\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242404 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-podinfo\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.242543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242460 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-envoy\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.242543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242519 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-credential-socket\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.242543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242528 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-socket" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242563 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-data\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242605 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-token\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-token\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242635 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-socket\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242665 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-certs\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242706 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-certs\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242745 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-podinfo\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242782 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istiod-ca-cert\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242840 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-token\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-token\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.242905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242877 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-kube-api-access-h2twn\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-kube-api-access-h2twn\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242920 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istiod-ca-cert\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.242977 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-kube-api-access-qcg98\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-kube-api-access-qcg98\") pod \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\" (UID: \"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0\") " Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243033 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-data\") pod \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\" (UID: \"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117\") " Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243304 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-socket" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243334 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-data" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243537 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-data" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243726 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-credential-socket" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243959 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69134f35-56c1-4e30-95e2-8989973e6a95-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243962 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-certs" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243980 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.243997 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244010 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244024 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244037 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244051 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244064 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244078 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244091 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244104 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69134f35-56c1-4e30-95e2-8989973e6a95-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244105 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-credential-socket" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244118 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244132 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244146 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244159 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244172 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244186 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244201 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244214 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244228 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69134f35-56c1-4e30-95e2-8989973e6a95-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244242 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvzg\" (UniqueName: \"kubernetes.io/projected/69134f35-56c1-4e30-95e2-8989973e6a95-kube-api-access-8vvzg\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244258 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxk8\" (UniqueName: \"kubernetes.io/projected/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27-kube-api-access-tdxk8\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244272 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.245163 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.244408 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istiod-ca-cert" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:17.247159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.246500 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istiod-ca-cert" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:17.247159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.246517 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-kube-api-access-h2twn" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "kube-api-access-h2twn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.247159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.246714 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-certs" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.247861 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.247666 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-kube-api-access-qcg98" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "kube-api-access-qcg98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.247861 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.247754 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-podinfo" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:17.247861 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.247830 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-envoy" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.247861 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.247851 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-envoy" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.248279 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.248254 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-token" pod "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" (UID: "f360b1bc-bbe1-4705-ae6b-b0d8de2fc117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.249244 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.249218 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-podinfo" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:17.252393 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.252364 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-token" pod "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" (UID: "6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.348147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.347997 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348040 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348055 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348069 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348085 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2twn\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-kube-api-access-h2twn\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348147 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348101 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348629 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcg98\" (UniqueName: \"kubernetes.io/projected/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-kube-api-access-qcg98\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348664 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348681 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348697 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348711 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348726 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.348763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.348741 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.734202 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.734167 4898 generic.go:356] "Generic (PLEG): container finished" podID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerID="66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849" exitCode=0 Nov 03 22:55:17.734389 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.734218 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" Nov 03 22:55:17.734389 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.734259 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" event={"ID":"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117","Type":"ContainerDied","Data":"66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849"} Nov 03 22:55:17.734389 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.734324 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw" event={"ID":"f360b1bc-bbe1-4705-ae6b-b0d8de2fc117","Type":"ContainerDied","Data":"0eab286540bbb3f220cd350c68b306ee8f34592f46356f104110d455f9a56c45"} Nov 03 22:55:17.734389 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.734359 4898 scope.go:122] "RemoveContainer" containerID="66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849" Nov 03 22:55:17.735916 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.735891 4898 generic.go:356] "Generic (PLEG): container finished" podID="69134f35-56c1-4e30-95e2-8989973e6a95" containerID="1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86" exitCode=0 Nov 03 22:55:17.736057 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.735963 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" Nov 03 22:55:17.736057 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.735983 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" event={"ID":"69134f35-56c1-4e30-95e2-8989973e6a95","Type":"ContainerDied","Data":"1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86"} Nov 03 22:55:17.736057 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.736036 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt" event={"ID":"69134f35-56c1-4e30-95e2-8989973e6a95","Type":"ContainerDied","Data":"f2e5fd93cbd3d94b6da6cae688e1806fe1f274ec9bba46a70eded7cdff55f4d7"} Nov 03 22:55:17.737473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.737450 4898 generic.go:356] "Generic (PLEG): container finished" podID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerID="522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403" exitCode=0 Nov 03 22:55:17.737594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.737496 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" Nov 03 22:55:17.737594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.737509 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" event={"ID":"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0","Type":"ContainerDied","Data":"522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403"} Nov 03 22:55:17.737594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.737581 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm" event={"ID":"6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0","Type":"ContainerDied","Data":"0ec086f1483f49e139185c370b4705ee3a7812a9494edbdd721d8e363b66166c"} Nov 03 22:55:17.739200 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.739176 4898 generic.go:356] "Generic (PLEG): container finished" podID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerID="296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02" exitCode=0 Nov 03 22:55:17.739309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.739222 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" Nov 03 22:55:17.739309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.739252 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" event={"ID":"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27","Type":"ContainerDied","Data":"296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02"} Nov 03 22:55:17.739309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.739301 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45" event={"ID":"aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27","Type":"ContainerDied","Data":"35cacbcce9d8374f11b782663dee7a4ada2f8a7085f8b5eefb4877b755f82581"} Nov 03 22:55:17.749286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.749171 4898 scope.go:122] "RemoveContainer" containerID="66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849" Nov 03 22:55:17.749555 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:17.749533 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849\": container with ID starting with 66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849 not found: ID does not exist" containerID="66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849" Nov 03 22:55:17.749621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.749571 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849"} err="failed to get container status \"66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849\": rpc error: code = NotFound desc = could not find container \"66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849\": container with ID starting with 66352f599071835e130ba5be469837a8d1ddded5ed78de3ac45559a091e77849 not found: ID does not exist" Nov 03 22:55:17.749621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.749599 4898 scope.go:122] "RemoveContainer" containerID="1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86" Nov 03 22:55:17.760089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.760069 4898 scope.go:122] "RemoveContainer" containerID="1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86" Nov 03 22:55:17.760365 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:17.760347 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86\": container with ID starting with 1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86 not found: ID does not exist" containerID="1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86" Nov 03 22:55:17.760428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.760373 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86"} err="failed to get container status \"1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86\": rpc error: code = NotFound desc = could not find container \"1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86\": container with ID starting with 1ecde121981d9450321cf96294d4238846dc444a865da8a4ccadb53d0c96ea86 not found: ID does not exist" Nov 03 22:55:17.760428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.760391 4898 scope.go:122] "RemoveContainer" containerID="522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403" Nov 03 22:55:17.771733 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.771694 4898 scope.go:122] "RemoveContainer" containerID="522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403" Nov 03 22:55:17.772359 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:17.772335 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403\": container with ID starting with 522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403 not found: ID does not exist" containerID="522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403" Nov 03 22:55:17.772575 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.772530 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403"} err="failed to get container status \"522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403\": rpc error: code = NotFound desc = could not find container \"522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403\": container with ID starting with 522597137ec63a941eb11d90127fae104cb6824e6dff3d6e63137411c0db9403 not found: ID does not exist" Nov 03 22:55:17.772724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.772710 4898 scope.go:122] "RemoveContainer" containerID="296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02" Nov 03 22:55:17.782836 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.782812 4898 scope.go:122] "RemoveContainer" containerID="296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02" Nov 03 22:55:17.783120 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:17.783097 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02\": container with ID starting with 296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02 not found: ID does not exist" containerID="296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02" Nov 03 22:55:17.783211 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.783129 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02"} err="failed to get container status \"296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02\": rpc error: code = NotFound desc = could not find container \"296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02\": container with ID starting with 296f699ae21c322106e9917a3225bb39b6015b5803e72b105a1946407aefcd02 not found: ID does not exist" Nov 03 22:55:17.987145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:17.987062 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45"] Nov 03 22:55:18.010086 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.010033 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ddv45"] Nov 03 22:55:18.129317 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.129288 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt"] Nov 03 22:55:18.134260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.134233 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n5gnt"] Nov 03 22:55:18.240397 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.240307 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw"] Nov 03 22:55:18.246196 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.246167 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fkwnw"] Nov 03 22:55:18.355387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.355354 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm"] Nov 03 22:55:18.360471 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.360445 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k2jmm"] Nov 03 22:55:18.908519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.908486 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" path="/var/lib/kubelet/pods/69134f35-56c1-4e30-95e2-8989973e6a95/volumes" Nov 03 22:55:18.908992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.908978 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" path="/var/lib/kubelet/pods/6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0/volumes" Nov 03 22:55:18.909407 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.909394 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" path="/var/lib/kubelet/pods/aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27/volumes" Nov 03 22:55:18.910130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:18.910099 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" path="/var/lib/kubelet/pods/f360b1bc-bbe1-4705-ae6b-b0d8de2fc117/volumes" Nov 03 22:55:19.836214 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:19.836173 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:55:29.391506 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.391470 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2"] Nov 03 22:55:29.392171 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392154 4898 cpu_manager.go:414] "containerMap: removing container" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" Nov 03 22:55:29.392218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392172 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" Nov 03 22:55:29.392218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392201 4898 cpu_manager.go:414] "containerMap: removing container" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" Nov 03 22:55:29.392218 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392206 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392218 4898 cpu_manager.go:414] "containerMap: removing container" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392223 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392239 4898 cpu_manager.go:414] "containerMap: removing container" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392244 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392254 4898 cpu_manager.go:414] "containerMap: removing container" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392258 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392267 4898 cpu_manager.go:414] "containerMap: removing container" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" Nov 03 22:55:29.392310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392271 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" Nov 03 22:55:29.392534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392363 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df6004b-a782-479f-9966-9c38ad4ed0a7" containerName="istio-proxy" Nov 03 22:55:29.392534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392370 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa713a28-16bb-48bc-8ba6-e4cbfdb2bd27" containerName="istio-proxy" Nov 03 22:55:29.392534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392381 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd03b3e-e4ce-4fdd-9baf-ca09a78c6df0" containerName="istio-proxy" Nov 03 22:55:29.392534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392390 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="636b01e1-8ade-44d2-af06-5c85804832e7" containerName="istio-proxy" Nov 03 22:55:29.392534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392396 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="69134f35-56c1-4e30-95e2-8989973e6a95" containerName="istio-proxy" Nov 03 22:55:29.392534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.392405 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f360b1bc-bbe1-4705-ae6b-b0d8de2fc117" containerName="istio-proxy" Nov 03 22:55:29.394775 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.394759 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.415260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.415232 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-metrics-certs\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.415480 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.415412 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96bc\" (UniqueName: \"kubernetes.io/projected/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-kube-api-access-b96bc\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.415585 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.415560 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.415710 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.415679 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.447587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.447563 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d6296b-serving-cert\"" Nov 03 22:55:29.501442 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.501408 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2"] Nov 03 22:55:29.522003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.516783 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.522003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.516856 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-metrics-certs\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.522003 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:29.516920 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-serving-cert: secret "maas-api-e2e-api-b-d6296b-serving-cert" not found Nov 03 22:55:29.522003 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:29.516990 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls podName:c7fb8b42-5285-4695-9cfa-cb4214b3ba03 nodeName:}" failed. No retries permitted until 2026-11-03 22:55:30.016971653 +0000 UTC m=+1973.229740699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls") pod "maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" (UID: "c7fb8b42-5285-4695-9cfa-cb4214b3ba03") : secret "maas-api-e2e-api-b-d6296b-serving-cert" not found Nov 03 22:55:29.522003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.517044 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96bc\" (UniqueName: \"kubernetes.io/projected/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-kube-api-access-b96bc\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.522003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.517151 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.522003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.518194 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.523622 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.523592 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-metrics-certs\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:29.581179 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:29.581140 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96bc\" (UniqueName: \"kubernetes.io/projected/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-kube-api-access-b96bc\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:30.023120 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:30.023084 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:30.025574 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:30.025543 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls\") pod \"maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:30.308486 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:30.308453 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:30.695771 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:55:30.695736 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fb8b42_5285_4695_9cfa_cb4214b3ba03.slice/crio-5d2a1c53c7b1534c276fd485671909d0663ff872226e11c0be76c83a56126511 WatchSource:0}: Error finding container 5d2a1c53c7b1534c276fd485671909d0663ff872226e11c0be76c83a56126511: Status 404 returned error can't find the container with id 5d2a1c53c7b1534c276fd485671909d0663ff872226e11c0be76c83a56126511 Nov 03 22:55:30.696234 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:30.696074 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2"] Nov 03 22:55:30.697470 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:30.697450 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:55:30.831707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:30.831663 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" event={"ID":"c7fb8b42-5285-4695-9cfa-cb4214b3ba03","Type":"ContainerStarted","Data":"5d2a1c53c7b1534c276fd485671909d0663ff872226e11c0be76c83a56126511"} Nov 03 22:55:31.843261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:31.843198 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" event={"ID":"c7fb8b42-5285-4695-9cfa-cb4214b3ba03","Type":"ContainerStarted","Data":"0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962"} Nov 03 22:55:31.843741 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:31.843287 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:31.964856 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:31.964742 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" podStartSLOduration=2.052710549 podStartE2EDuration="2.964728298s" totalImagesPullingTime="912.017749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:31.961042907 +0000 UTC m=+1975.173811958" watchObservedRunningTime="2026-11-03 22:55:31.964728298 +0000 UTC m=+1975.177497386" Nov 03 22:55:37.855280 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:37.855241 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 22:55:49.837864 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:49.837774 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:55:50.060295 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.060252 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc"] Nov 03 22:55:50.071473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.071437 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.116834 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.116752 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc"] Nov 03 22:55:50.163718 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.163682 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9674f5-serving-cert\"" Nov 03 22:55:50.192447 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.192401 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c89e2d9-6024-479f-8022-6bdda00c9c02-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.192628 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.192572 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.192694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.192645 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd28m\" (UniqueName: \"kubernetes.io/projected/6c89e2d9-6024-479f-8022-6bdda00c9c02-kube-api-access-hd28m\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.192694 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.192687 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-metrics-certs\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.294190 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.294151 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c89e2d9-6024-479f-8022-6bdda00c9c02-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.294370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.294255 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.294370 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.294335 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd28m\" (UniqueName: \"kubernetes.io/projected/6c89e2d9-6024-479f-8022-6bdda00c9c02-kube-api-access-hd28m\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.294504 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:50.294366 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-serving-cert: secret "maas-api-e2e-inf-b-9674f5-serving-cert" not found Nov 03 22:55:50.294504 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:55:50.294435 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls podName:6c89e2d9-6024-479f-8022-6bdda00c9c02 nodeName:}" failed. No retries permitted until 2026-11-03 22:55:50.794418605 +0000 UTC m=+1994.007187633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls") pod "maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" (UID: "6c89e2d9-6024-479f-8022-6bdda00c9c02") : secret "maas-api-e2e-inf-b-9674f5-serving-cert" not found Nov 03 22:55:50.294504 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.294373 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-metrics-certs\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.295008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.294988 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c89e2d9-6024-479f-8022-6bdda00c9c02-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.296974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.296950 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-metrics-certs\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.361083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.361049 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd28m\" (UniqueName: \"kubernetes.io/projected/6c89e2d9-6024-479f-8022-6bdda00c9c02-kube-api-access-hd28m\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.801224 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.801184 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.803736 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.803702 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls\") pod \"maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:50.984967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:50.984931 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:51.370310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:51.370276 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc"] Nov 03 22:55:51.371419 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:55:51.371389 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c89e2d9_6024_479f_8022_6bdda00c9c02.slice/crio-c2dc45bee9de971dfbbe002a3213e716c89af76a674c02a56a595bb7d10a67e8 WatchSource:0}: Error finding container c2dc45bee9de971dfbbe002a3213e716c89af76a674c02a56a595bb7d10a67e8: Status 404 returned error can't find the container with id c2dc45bee9de971dfbbe002a3213e716c89af76a674c02a56a595bb7d10a67e8 Nov 03 22:55:51.987080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:51.987031 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" event={"ID":"6c89e2d9-6024-479f-8022-6bdda00c9c02","Type":"ContainerStarted","Data":"c2dc45bee9de971dfbbe002a3213e716c89af76a674c02a56a595bb7d10a67e8"} Nov 03 22:55:52.995534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:52.995491 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" event={"ID":"6c89e2d9-6024-479f-8022-6bdda00c9c02","Type":"ContainerStarted","Data":"3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a"} Nov 03 22:55:52.996072 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:52.995586 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:55:53.116475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:53.116416 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" podStartSLOduration=3.241509228 podStartE2EDuration="4.11640107s" totalImagesPullingTime="874.891842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:53.114129637 +0000 UTC m=+1996.326898712" watchObservedRunningTime="2026-11-03 22:55:53.11640107 +0000 UTC m=+1996.329170122" Nov 03 22:55:59.015327 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:55:59.015299 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 22:56:12.144869 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:12.144822 4898 generic.go:356] "Generic (PLEG): container finished" podID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerID="9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04" exitCode=0 Nov 03 22:56:12.145405 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:12.144905 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" event={"ID":"4d2267cf-8669-41cd-acbd-8f02e561853e","Type":"ContainerDied","Data":"9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04"} Nov 03 22:56:12.145474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:12.145447 4898 scope.go:122] "RemoveContainer" containerID="9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04" Nov 03 22:56:12.147336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:12.147307 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:12.147474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:12.147380 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:12.147474 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:12.147413 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:13.172041 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:13.172000 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" event={"ID":"4d2267cf-8669-41cd-acbd-8f02e561853e","Type":"ContainerStarted","Data":"3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d"} Nov 03 22:56:13.363812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:13.363170 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:56:13.368975 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:13.368936 4898 patch_prober.go:29] interesting pod/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:56:13.369140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:13.369004 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:56:14.124607 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.124572 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq"] Nov 03 22:56:14.124822 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.124765 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" podUID="88d97b63-31a2-40ac-b978-2da735c4eec7" containerName="maas-api" containerID="cri-o://f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa" gracePeriod=30 Nov 03 22:56:14.371996 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.363210 4898 patch_prober.go:29] interesting pod/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:56:14.371996 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.363270 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:56:14.782837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.781844 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:56:14.894472 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.894433 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d97b63-31a2-40ac-b978-2da735c4eec7-kube-api-access-2xc4m\" (UniqueName: \"kubernetes.io/projected/88d97b63-31a2-40ac-b978-2da735c4eec7-kube-api-access-2xc4m\") pod \"88d97b63-31a2-40ac-b978-2da735c4eec7\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " Nov 03 22:56:14.894675 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.894619 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls\") pod \"88d97b63-31a2-40ac-b978-2da735c4eec7\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " Nov 03 22:56:14.894675 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.894648 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-metrics-certs\") pod \"88d97b63-31a2-40ac-b978-2da735c4eec7\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " Nov 03 22:56:14.894829 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.894779 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d97b63-31a2-40ac-b978-2da735c4eec7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d97b63-31a2-40ac-b978-2da735c4eec7-openshift-service-ca\") pod \"88d97b63-31a2-40ac-b978-2da735c4eec7\" (UID: \"88d97b63-31a2-40ac-b978-2da735c4eec7\") " Nov 03 22:56:14.895186 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.895153 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d97b63-31a2-40ac-b978-2da735c4eec7-openshift-service-ca" pod "88d97b63-31a2-40ac-b978-2da735c4eec7" (UID: "88d97b63-31a2-40ac-b978-2da735c4eec7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:14.895305 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.895281 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d97b63-31a2-40ac-b978-2da735c4eec7-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:14.896667 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.896648 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d97b63-31a2-40ac-b978-2da735c4eec7-kube-api-access-2xc4m" pod "88d97b63-31a2-40ac-b978-2da735c4eec7" (UID: "88d97b63-31a2-40ac-b978-2da735c4eec7"). InnerVolumeSpecName "kube-api-access-2xc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:14.896743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.896723 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls" pod "88d97b63-31a2-40ac-b978-2da735c4eec7" (UID: "88d97b63-31a2-40ac-b978-2da735c4eec7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:14.896843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.896828 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-metrics-certs" pod "88d97b63-31a2-40ac-b978-2da735c4eec7" (UID: "88d97b63-31a2-40ac-b978-2da735c4eec7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:14.996875 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.996750 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:14.996875 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.996800 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d97b63-31a2-40ac-b978-2da735c4eec7-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:14.996875 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:14.996818 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xc4m\" (UniqueName: \"kubernetes.io/projected/88d97b63-31a2-40ac-b978-2da735c4eec7-kube-api-access-2xc4m\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:15.199983 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.199943 4898 generic.go:356] "Generic (PLEG): container finished" podID="88d97b63-31a2-40ac-b978-2da735c4eec7" containerID="f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa" exitCode=0 Nov 03 22:56:15.200135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.200005 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" Nov 03 22:56:15.200135 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.200106 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" event={"ID":"88d97b63-31a2-40ac-b978-2da735c4eec7","Type":"ContainerDied","Data":"f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa"} Nov 03 22:56:15.200235 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.200142 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq" event={"ID":"88d97b63-31a2-40ac-b978-2da735c4eec7","Type":"ContainerDied","Data":"1c3be47a9c5d416ddf648e5bc26cf5c0c92ccc38092191297dfff5de621b07db"} Nov 03 22:56:15.200235 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.200163 4898 scope.go:122] "RemoveContainer" containerID="f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa" Nov 03 22:56:15.222548 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.222516 4898 scope.go:122] "RemoveContainer" containerID="f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa" Nov 03 22:56:15.222961 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:15.222934 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa\": container with ID starting with f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa not found: ID does not exist" containerID="f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa" Nov 03 22:56:15.223172 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.223084 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa"} err="failed to get container status \"f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa\": rpc error: code = NotFound desc = could not find container \"f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa\": container with ID starting with f824ca471666b5be01d4f0cb8ab9d58d62775dc4c8f0a5cdec5016562aa6f2fa not found: ID does not exist" Nov 03 22:56:15.307828 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.307744 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx"] Nov 03 22:56:15.307992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.307971 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" podUID="c960bcf8-805c-4f86-a398-b476ad90ea44" containerName="maas-api" containerID="cri-o://1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe" gracePeriod=30 Nov 03 22:56:15.373846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.369455 4898 patch_prober.go:29] interesting pod/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:56:15.373846 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.369516 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:56:15.387803 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.387752 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq"] Nov 03 22:56:15.391924 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.391896 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922795-566c9947bc-24czq"] Nov 03 22:56:15.526926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.526892 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:56:15.896355 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:15.896277 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:56:16.023854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.021217 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-metrics-certs\") pod \"c960bcf8-805c-4f86-a398-b476ad90ea44\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " Nov 03 22:56:16.023854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.021300 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls\") pod \"c960bcf8-805c-4f86-a398-b476ad90ea44\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " Nov 03 22:56:16.023854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.021343 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c960bcf8-805c-4f86-a398-b476ad90ea44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c960bcf8-805c-4f86-a398-b476ad90ea44-openshift-service-ca\") pod \"c960bcf8-805c-4f86-a398-b476ad90ea44\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " Nov 03 22:56:16.023854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.021432 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c960bcf8-805c-4f86-a398-b476ad90ea44-kube-api-access-stmpl\" (UniqueName: \"kubernetes.io/projected/c960bcf8-805c-4f86-a398-b476ad90ea44-kube-api-access-stmpl\") pod \"c960bcf8-805c-4f86-a398-b476ad90ea44\" (UID: \"c960bcf8-805c-4f86-a398-b476ad90ea44\") " Nov 03 22:56:16.023854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.021848 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c960bcf8-805c-4f86-a398-b476ad90ea44-openshift-service-ca" pod "c960bcf8-805c-4f86-a398-b476ad90ea44" (UID: "c960bcf8-805c-4f86-a398-b476ad90ea44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:16.023854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.022003 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c960bcf8-805c-4f86-a398-b476ad90ea44-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:16.028191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.026244 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls" pod "c960bcf8-805c-4f86-a398-b476ad90ea44" (UID: "c960bcf8-805c-4f86-a398-b476ad90ea44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:16.028191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.026258 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c960bcf8-805c-4f86-a398-b476ad90ea44-kube-api-access-stmpl" pod "c960bcf8-805c-4f86-a398-b476ad90ea44" (UID: "c960bcf8-805c-4f86-a398-b476ad90ea44"). InnerVolumeSpecName "kube-api-access-stmpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:16.028191 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.028049 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-metrics-certs" pod "c960bcf8-805c-4f86-a398-b476ad90ea44" (UID: "c960bcf8-805c-4f86-a398-b476ad90ea44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:16.123423 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.123390 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:16.123423 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.123418 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c960bcf8-805c-4f86-a398-b476ad90ea44-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:16.123423 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.123428 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stmpl\" (UniqueName: \"kubernetes.io/projected/c960bcf8-805c-4f86-a398-b476ad90ea44-kube-api-access-stmpl\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:16.208457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.208369 4898 generic.go:356] "Generic (PLEG): container finished" podID="c960bcf8-805c-4f86-a398-b476ad90ea44" containerID="1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe" exitCode=0 Nov 03 22:56:16.208457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.208418 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" Nov 03 22:56:16.208665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.208457 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" event={"ID":"c960bcf8-805c-4f86-a398-b476ad90ea44","Type":"ContainerDied","Data":"1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe"} Nov 03 22:56:16.208665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.208511 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx" event={"ID":"c960bcf8-805c-4f86-a398-b476ad90ea44","Type":"ContainerDied","Data":"d97ce3f67c2747a046459248529a1d3ed69c90af0a3f5af585d4e241832f2392"} Nov 03 22:56:16.208665 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.208543 4898 scope.go:122] "RemoveContainer" containerID="1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe" Nov 03 22:56:16.220246 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.220221 4898 scope.go:122] "RemoveContainer" containerID="1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe" Nov 03 22:56:16.220547 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:16.220529 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe\": container with ID starting with 1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe not found: ID does not exist" containerID="1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe" Nov 03 22:56:16.220628 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.220560 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe"} err="failed to get container status \"1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe\": rpc error: code = NotFound desc = could not find container \"1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe\": container with ID starting with 1aba66a02738e179c9c61d726ebe42e0089f6ab113f9e55e285a47e69dcdefbe not found: ID does not exist" Nov 03 22:56:16.370213 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.370173 4898 patch_prober.go:29] interesting pod/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:56:16.370497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.370463 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:56:16.394488 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.394455 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx"] Nov 03 22:56:16.402843 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.402809 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb4f0a-5474cddb99-9m2tx"] Nov 03 22:56:16.907124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.907041 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d97b63-31a2-40ac-b978-2da735c4eec7" path="/var/lib/kubelet/pods/88d97b63-31a2-40ac-b978-2da735c4eec7/volumes" Nov 03 22:56:16.907502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:16.907487 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c960bcf8-805c-4f86-a398-b476ad90ea44" path="/var/lib/kubelet/pods/c960bcf8-805c-4f86-a398-b476ad90ea44/volumes" Nov 03 22:56:17.365233 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:17.365194 4898 patch_prober.go:29] interesting pod/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:56:17.365476 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:17.365262 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:56:18.363416 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:18.363376 4898 patch_prober.go:29] interesting pod/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:56:18.363917 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:18.363445 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:56:19.368518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:19.368484 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:56:19.369638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:19.369612 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 22:56:19.838802 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:19.838751 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:56:22.325403 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.325371 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5"] Nov 03 22:56:22.326069 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.326055 4898 cpu_manager.go:414] "containerMap: removing container" podUID="c960bcf8-805c-4f86-a398-b476ad90ea44" containerName="maas-api" Nov 03 22:56:22.326114 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.326095 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c960bcf8-805c-4f86-a398-b476ad90ea44" containerName="maas-api" Nov 03 22:56:22.326157 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.326148 4898 cpu_manager.go:414] "containerMap: removing container" podUID="88d97b63-31a2-40ac-b978-2da735c4eec7" containerName="maas-api" Nov 03 22:56:22.326197 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.326156 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d97b63-31a2-40ac-b978-2da735c4eec7" containerName="maas-api" Nov 03 22:56:22.326261 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.326253 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d97b63-31a2-40ac-b978-2da735c4eec7" containerName="maas-api" Nov 03 22:56:22.326301 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.326267 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c960bcf8-805c-4f86-a398-b476ad90ea44" containerName="maas-api" Nov 03 22:56:22.331272 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.331256 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.386197 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.386168 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5"] Nov 03 22:56:22.425892 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.425853 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.426063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.425990 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-metrics-certs\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.426159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.426118 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptblc\" (UniqueName: \"kubernetes.io/projected/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-kube-api-access-ptblc\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.426250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.426234 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-openshift-service-ca\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.433176 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.433150 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-12450a06-serving-cert\"" Nov 03 22:56:22.527187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.527156 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.527363 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.527212 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-metrics-certs\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.527363 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.527258 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptblc\" (UniqueName: \"kubernetes.io/projected/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-kube-api-access-ptblc\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.527363 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:22.527294 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-serving-cert: secret "maas-api-e2e-mt-12450a06-serving-cert" not found Nov 03 22:56:22.527363 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.527315 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-openshift-service-ca\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.527566 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:22.527375 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls podName:b73a40f4-2328-4de8-9ec5-569bd38d8f3b nodeName:}" failed. No retries permitted until 2026-11-03 22:56:23.027354676 +0000 UTC m=+2026.240123733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls") pod "maas-api-e2e-mt-12450a06-68776f84c-k6kr5" (UID: "b73a40f4-2328-4de8-9ec5-569bd38d8f3b") : secret "maas-api-e2e-mt-12450a06-serving-cert" not found Nov 03 22:56:22.528090 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.528064 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-openshift-service-ca\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.529821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.529778 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-metrics-certs\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:22.595448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:22.595380 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptblc\" (UniqueName: \"kubernetes.io/projected/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-kube-api-access-ptblc\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:23.034895 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:23.034814 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:23.037426 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:23.037401 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls\") pod \"maas-api-e2e-mt-12450a06-68776f84c-k6kr5\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:23.210430 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:23.210390 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47"] Nov 03 22:56:23.210754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:23.210721 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" containerID="cri-o://819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27" gracePeriod=30 Nov 03 22:56:23.245534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:23.245498 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:23.683316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:23.683287 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5"] Nov 03 22:56:23.684336 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:56:23.684307 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73a40f4_2328_4de8_9ec5_569bd38d8f3b.slice/crio-0f9220be72d3459b322a8316710994ea0726c7b2f2628589c66b48eebf9d4ff8 WatchSource:0}: Error finding container 0f9220be72d3459b322a8316710994ea0726c7b2f2628589c66b48eebf9d4ff8: Status 404 returned error can't find the container with id 0f9220be72d3459b322a8316710994ea0726c7b2f2628589c66b48eebf9d4ff8 Nov 03 22:56:24.285390 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:24.285352 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" event={"ID":"b73a40f4-2328-4de8-9ec5-569bd38d8f3b","Type":"ContainerStarted","Data":"0f9220be72d3459b322a8316710994ea0726c7b2f2628589c66b48eebf9d4ff8"} Nov 03 22:56:25.294766 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:25.294729 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" event={"ID":"b73a40f4-2328-4de8-9ec5-569bd38d8f3b","Type":"ContainerStarted","Data":"c8163005d1988ba68f157cca39125c14ebc12c676646943ef87e738936641bb1"} Nov 03 22:56:25.295216 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:25.294943 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:25.423108 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:25.423047 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" podStartSLOduration=2.478696706 podStartE2EDuration="3.423028101s" totalImagesPullingTime="944.331395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:25.421743228 +0000 UTC m=+2028.634512280" watchObservedRunningTime="2026-11-03 22:56:25.423028101 +0000 UTC m=+2028.635797153" Nov 03 22:56:28.572926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.572896 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:56:28.668165 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668126 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba948794-2afd-408b-bd73-b7f1d520f746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba948794-2afd-408b-bd73-b7f1d520f746-istio-podinfo\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668194 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-credential-socket\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668233 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-envoy\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668262 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-certs\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668285 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba948794-2afd-408b-bd73-b7f1d520f746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba948794-2afd-408b-bd73-b7f1d520f746-istiod-ca-cert\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668338 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668312 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-istio-token\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-istio-token\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668505 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668344 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-socket\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668505 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668402 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-kube-api-access-r9wkz\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-kube-api-access-r9wkz\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.668505 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.668429 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-data\") pod \"ba948794-2afd-408b-bd73-b7f1d520f746\" (UID: \"ba948794-2afd-408b-bd73-b7f1d520f746\") " Nov 03 22:56:28.669062 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.669028 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-credential-socket" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:28.669280 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.669048 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-certs" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:28.669402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.669178 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-socket" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:28.669495 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.669281 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba948794-2afd-408b-bd73-b7f1d520f746-istiod-ca-cert" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:28.669546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.669344 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-data" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:28.671098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.671070 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-envoy" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:28.671294 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.671273 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-kube-api-access-r9wkz" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "kube-api-access-r9wkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:28.672170 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.672148 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba948794-2afd-408b-bd73-b7f1d520f746-istio-podinfo" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:28.672284 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.672264 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-istio-token" pod "ba948794-2afd-408b-bd73-b7f1d520f746" (UID: "ba948794-2afd-408b-bd73-b7f1d520f746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:28.770970 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.770935 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.770970 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.770972 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.771452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.770987 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.771452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.771005 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba948794-2afd-408b-bd73-b7f1d520f746-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.771452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.771019 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.771452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.771033 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.771452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.771048 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9wkz\" (UniqueName: \"kubernetes.io/projected/ba948794-2afd-408b-bd73-b7f1d520f746-kube-api-access-r9wkz\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.771452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.771064 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba948794-2afd-408b-bd73-b7f1d520f746-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:28.771452 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:28.771077 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba948794-2afd-408b-bd73-b7f1d520f746-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:29.326168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.326131 4898 generic.go:356] "Generic (PLEG): container finished" podID="ba948794-2afd-408b-bd73-b7f1d520f746" containerID="819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27" exitCode=0 Nov 03 22:56:29.326362 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.326190 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" Nov 03 22:56:29.326362 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.326232 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" event={"ID":"ba948794-2afd-408b-bd73-b7f1d520f746","Type":"ContainerDied","Data":"819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27"} Nov 03 22:56:29.326362 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.326313 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47" event={"ID":"ba948794-2afd-408b-bd73-b7f1d520f746","Type":"ContainerDied","Data":"d080ec3f121714fe9e0eb881a8f2acccc40b1df434a46b3ba7df98b9850c7236"} Nov 03 22:56:29.326362 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.326360 4898 scope.go:122] "RemoveContainer" containerID="819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27" Nov 03 22:56:29.367110 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.366947 4898 scope.go:122] "RemoveContainer" containerID="014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018" Nov 03 22:56:29.381973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.381944 4898 scope.go:122] "RemoveContainer" containerID="819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27" Nov 03 22:56:29.385244 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:29.385198 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27\": container with ID starting with 819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27 not found: ID does not exist" containerID="819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27" Nov 03 22:56:29.385347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.385254 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27"} err="failed to get container status \"819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27\": rpc error: code = NotFound desc = could not find container \"819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27\": container with ID starting with 819b571b87ed80b8a979fc6c4015ec3214f51cf6edb31ff3fc3b6ce5dfe6ee27 not found: ID does not exist" Nov 03 22:56:29.385347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.385278 4898 scope.go:122] "RemoveContainer" containerID="014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018" Nov 03 22:56:29.389821 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:29.386935 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018\": container with ID starting with 014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018 not found: ID does not exist" containerID="014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018" Nov 03 22:56:29.389821 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.386971 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018"} err="failed to get container status \"014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018\": rpc error: code = NotFound desc = could not find container \"014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018\": container with ID starting with 014936a705c40ddc9f6d1b722d1e11171ea55fbc66351e75b6e2df29d33c5018 not found: ID does not exist" Nov 03 22:56:29.497252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.497212 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47"] Nov 03 22:56:29.555614 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:29.555576 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922795-openshift-default-59b46784df-6bc47"] Nov 03 22:56:30.907183 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:30.907144 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" path="/var/lib/kubelet/pods/ba948794-2afd-408b-bd73-b7f1d520f746/volumes" Nov 03 22:56:31.308568 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:31.308489 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:56:32.174422 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.174384 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md"] Nov 03 22:56:32.174981 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.174597 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" podUID="4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" containerName="maas-api" containerID="cri-o://39fc664c8c197b575246d825ee4f502fcd34d5481c979925abafea324933bad9" gracePeriod=30 Nov 03 22:56:32.386660 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.386624 4898 generic.go:356] "Generic (PLEG): container finished" podID="4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" containerID="39fc664c8c197b575246d825ee4f502fcd34d5481c979925abafea324933bad9" exitCode=0 Nov 03 22:56:32.386873 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.386750 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" event={"ID":"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b","Type":"ContainerDied","Data":"39fc664c8c197b575246d825ee4f502fcd34d5481c979925abafea324933bad9"} Nov 03 22:56:32.572600 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.572571 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:56:32.629417 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.629383 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-maas-api-tls\") pod \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " Nov 03 22:56:32.629598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.629511 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-metrics-certs\") pod \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " Nov 03 22:56:32.629598 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.629570 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-openshift-service-ca\") pod \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " Nov 03 22:56:32.630554 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.629679 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-kube-api-access-qmgt8\" (UniqueName: \"kubernetes.io/projected/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-kube-api-access-qmgt8\") pod \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\" (UID: \"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b\") " Nov 03 22:56:32.630554 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.630050 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-openshift-service-ca" pod "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" (UID: "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:32.630554 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.630509 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:32.632475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.632446 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-metrics-certs" pod "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" (UID: "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:32.634260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.634233 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-kube-api-access-qmgt8" pod "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" (UID: "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b"). InnerVolumeSpecName "kube-api-access-qmgt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:32.642225 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.642188 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-maas-api-tls" pod "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" (UID: "4602fcaa-dbe8-4fd5-b02d-6326acb9d22b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:32.732155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.732076 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:32.732155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.732113 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:32.732155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:32.732129 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmgt8\" (UniqueName: \"kubernetes.io/projected/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b-kube-api-access-qmgt8\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:33.396612 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:33.396581 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" Nov 03 22:56:33.397064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:33.396580 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md" event={"ID":"4602fcaa-dbe8-4fd5-b02d-6326acb9d22b","Type":"ContainerDied","Data":"d90d25085cca44c7626cdbfcaddef1e09aa1c7e52ead32a1f767ee1dd8601a18"} Nov 03 22:56:33.397064 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:33.396725 4898 scope.go:122] "RemoveContainer" containerID="39fc664c8c197b575246d825ee4f502fcd34d5481c979925abafea324933bad9" Nov 03 22:56:33.565601 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:33.565512 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md"] Nov 03 22:56:33.615285 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:33.615245 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6a3169-5ccb6b5c8b-7j2md"] Nov 03 22:56:34.907291 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:34.907252 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" path="/var/lib/kubelet/pods/4602fcaa-dbe8-4fd5-b02d-6326acb9d22b/volumes" Nov 03 22:56:36.423237 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:36.423208 4898 generic.go:356] "Generic (PLEG): container finished" podID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerID="958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50" exitCode=0 Nov 03 22:56:36.423648 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:36.423282 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" event={"ID":"661c0a78-c7ec-4875-a439-fafe63b9c2d2","Type":"ContainerDied","Data":"958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50"} Nov 03 22:56:36.423732 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:36.423717 4898 scope.go:122] "RemoveContainer" containerID="958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50" Nov 03 22:56:36.425502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:36.425470 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:36.425631 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:36.425552 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:36.425631 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:36.425591 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:37.433275 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:37.433232 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" event={"ID":"661c0a78-c7ec-4875-a439-fafe63b9c2d2","Type":"ContainerStarted","Data":"26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401"} Nov 03 22:56:37.589997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:37.589957 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:56:37.590205 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:37.590137 4898 patch_prober.go:29] interesting pod/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:56:37.590205 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:37.590191 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 22:56:38.590032 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:38.589989 4898 patch_prober.go:29] interesting pod/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:56:38.590556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:38.590054 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 22:56:39.590184 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:39.590148 4898 patch_prober.go:29] interesting pod/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:56:39.590607 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:39.590208 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 22:56:40.593203 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:40.593154 4898 patch_prober.go:29] interesting pod/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:56:40.593738 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:40.593218 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 22:56:41.590207 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:41.590174 4898 patch_prober.go:29] interesting pod/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:56:41.590396 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:41.590227 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 22:56:42.688549 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:42.688462 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:56:42.689428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:42.689406 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:56:42.690255 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:56:42.690231 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 22:56:49.835099 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:56:49.835061 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:57:10.479441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.479400 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght"] Nov 03 22:57:10.480411 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.480390 4898 cpu_manager.go:414] "containerMap: removing container" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" Nov 03 22:57:10.480473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.480414 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" Nov 03 22:57:10.480473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.480432 4898 cpu_manager.go:414] "containerMap: removing container" podUID="4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" containerName="maas-api" Nov 03 22:57:10.480473 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.480439 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" containerName="maas-api" Nov 03 22:57:10.480646 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.480633 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" Nov 03 22:57:10.480684 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.480652 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4602fcaa-dbe8-4fd5-b02d-6326acb9d22b" containerName="maas-api" Nov 03 22:57:10.483295 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.483279 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.538397 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.538311 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-12450a06-openshift-default-dockercfg-wvwz8\"" Nov 03 22:57:10.552926 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.552895 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e576686a-090f-4836-b6e3-31024fc18440-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.553083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.552936 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktzz\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-kube-api-access-hktzz\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.553130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.553092 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.553192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.553166 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e576686a-090f-4836-b6e3-31024fc18440-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.553303 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.553272 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.553357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.553323 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.554599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.553456 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.554599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.553510 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.554599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.553608 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.623812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.621107 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght"] Nov 03 22:57:10.666972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.666935 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e576686a-090f-4836-b6e3-31024fc18440-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.666993 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktzz\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-kube-api-access-hktzz\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667072 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667153 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e576686a-090f-4836-b6e3-31024fc18440-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667214 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667242 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667340 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667389 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.667546 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667431 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.668013 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.667982 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.668919 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.668891 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e576686a-090f-4836-b6e3-31024fc18440-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.681812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.678588 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e576686a-090f-4836-b6e3-31024fc18440-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.681812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.678879 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.681812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.679062 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.681812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.679513 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.681812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.680204 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.733316 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.733237 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.733482 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.733423 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktzz\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-kube-api-access-hktzz\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-6zght\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:10.796716 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:10.796668 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:11.233209 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.233178 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght"] Nov 03 22:57:11.234153 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:57:11.234125 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode576686a_090f_4836_b6e3_31024fc18440.slice/crio-2871cf40561b9bae0a52dd47442d77eb45bf9a3f9a5a4765faa3394088c905f3 WatchSource:0}: Error finding container 2871cf40561b9bae0a52dd47442d77eb45bf9a3f9a5a4765faa3394088c905f3: Status 404 returned error can't find the container with id 2871cf40561b9bae0a52dd47442d77eb45bf9a3f9a5a4765faa3394088c905f3 Nov 03 22:57:11.237551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.237512 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:11.237633 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.237600 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:11.237673 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.237645 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:11.707521 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.707488 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" event={"ID":"e576686a-090f-4836-b6e3-31024fc18440","Type":"ContainerStarted","Data":"2ea2e045aeeed925eadb437115c413afad7f5cc34e22dfa28514eb8e02af652d"} Nov 03 22:57:11.707956 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.707538 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" event={"ID":"e576686a-090f-4836-b6e3-31024fc18440","Type":"ContainerStarted","Data":"2871cf40561b9bae0a52dd47442d77eb45bf9a3f9a5a4765faa3394088c905f3"} Nov 03 22:57:11.796781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.796739 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:11.831178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:11.831118 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" podStartSLOduration=1.831105056 podStartE2EDuration="1.831105056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:11.828320883 +0000 UTC m=+2075.041089934" watchObservedRunningTime="2026-11-03 22:57:11.831105056 +0000 UTC m=+2075.043874106" Nov 03 22:57:12.799070 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:12.799035 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-6zght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:57:12.799478 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:12.799087 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:57:13.797995 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:13.797959 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-6zght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:57:13.798180 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:13.798026 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:57:14.797621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:14.797579 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-6zght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:57:14.798320 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:14.797658 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:57:15.797975 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:15.797942 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-6zght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:57:15.798468 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:15.797993 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:57:16.797774 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:16.797744 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-6zght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:57:16.797968 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:16.797808 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:57:17.802046 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:17.802016 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:17.802527 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:17.802507 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:17.803164 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:17.803141 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 22:57:19.836263 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:57:19.836219 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:57:37.464967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:37.464938 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:57:37.469577 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:37.469555 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:57:37.510402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:37.510374 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 22:57:37.515375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:37.515355 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 22:57:40.327138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.327103 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz"] Nov 03 22:57:40.327625 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.327358 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" containerID="cri-o://be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02" gracePeriod=30 Nov 03 22:57:40.713220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.713123 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc"] Nov 03 22:57:40.714089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.714063 4898 cpu_manager.go:414] "containerMap: removing container" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" Nov 03 22:57:40.714089 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.714085 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" Nov 03 22:57:40.714230 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.714205 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba948794-2afd-408b-bd73-b7f1d520f746" containerName="istio-proxy" Nov 03 22:57:40.717226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.717186 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.775319 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.775285 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc"] Nov 03 22:57:40.781176 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.781144 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.781341 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.781187 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.781341 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.781274 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.781466 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.781355 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.781466 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.781431 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.781817 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.781778 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.782077 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.782054 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.782168 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.782093 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g4sn\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-kube-api-access-8g4sn\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.782238 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.782218 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45923fe-4cf8-4adb-93c8-b84d41155806-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884003 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.883941 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884030 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884062 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g4sn\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-kube-api-access-8g4sn\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884109 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45923fe-4cf8-4adb-93c8-b84d41155806-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884229 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884257 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884292 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884334 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884631 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884402 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884631 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884456 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884631 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884487 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884658 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.884811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.884749 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.885049 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.885030 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45923fe-4cf8-4adb-93c8-b84d41155806-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.886760 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.886736 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.886969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.886948 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.945358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.945324 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:40.945603 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:40.945578 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g4sn\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-kube-api-access-8g4sn\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:41.030638 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:41.030560 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:41.479441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:41.479409 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc"] Nov 03 22:57:41.480941 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:57:41.480912 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45923fe_4cf8_4adb_93c8_b84d41155806.slice/crio-873bbc43f89de42f5a19361b62a764367e8020b6f657b5908b895958fec463a8 WatchSource:0}: Error finding container 873bbc43f89de42f5a19361b62a764367e8020b6f657b5908b895958fec463a8: Status 404 returned error can't find the container with id 873bbc43f89de42f5a19361b62a764367e8020b6f657b5908b895958fec463a8 Nov 03 22:57:41.484061 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:41.484026 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:41.484136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:41.484092 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:41.484136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:41.484123 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:41.939645 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:41.939606 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" event={"ID":"a45923fe-4cf8-4adb-93c8-b84d41155806","Type":"ContainerStarted","Data":"a27e8f6d6863c79cdc1eddae63dcd0931ff10216336e2fd7258d5f4c0449556c"} Nov 03 22:57:41.939835 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:41.939671 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" event={"ID":"a45923fe-4cf8-4adb-93c8-b84d41155806","Type":"ContainerStarted","Data":"873bbc43f89de42f5a19361b62a764367e8020b6f657b5908b895958fec463a8"} Nov 03 22:57:42.033019 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:42.032961 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:42.040840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:42.037613 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:57:42.040840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:42.037677 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 22:57:42.066175 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:42.066107 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podStartSLOduration=2.066091491 podStartE2EDuration="2.066091491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:42.063275369 +0000 UTC m=+2105.276044450" watchObservedRunningTime="2026-11-03 22:57:42.066091491 +0000 UTC m=+2105.278860541" Nov 03 22:57:42.928210 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:42.928176 4898 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:57:42.928677 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:42.928235 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:57:43.030866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:43.030833 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:57:43.031071 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:43.030891 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 22:57:44.030951 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:44.030914 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:57:44.031416 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:44.030958 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 22:57:45.031429 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.031394 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:57:45.031894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.031452 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 22:57:45.670550 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.670522 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:57:45.770252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770210 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95aff2e6-4643-40f2-a714-46ec991854a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aff2e6-4643-40f2-a714-46ec991854a4-istiod-ca-cert\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770258 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-istio-token\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-istio-token\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770306 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-certs\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770349 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-envoy\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770376 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-credential-socket\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770464 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-kube-api-access-qvgg6\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-kube-api-access-qvgg6\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770524 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770518 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95aff2e6-4643-40f2-a714-46ec991854a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aff2e6-4643-40f2-a714-46ec991854a4-istio-podinfo\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770583 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-socket\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770623 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-data\") pod \"95aff2e6-4643-40f2-a714-46ec991854a4\" (UID: \"95aff2e6-4643-40f2-a714-46ec991854a4\") " Nov 03 22:57:45.770816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770637 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-certs" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:45.770816 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770693 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-credential-socket" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:45.771056 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.770968 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-socket" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:45.771116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.771101 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-data" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:45.771322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.771294 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.771322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.771325 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.771535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.771341 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.771535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.771354 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.771535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.771482 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95aff2e6-4643-40f2-a714-46ec991854a4-istiod-ca-cert" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:45.772733 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.772703 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-istio-token" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:45.772927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.772908 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-kube-api-access-qvgg6" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "kube-api-access-qvgg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:45.772996 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.772953 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95aff2e6-4643-40f2-a714-46ec991854a4-istio-podinfo" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:57:45.773041 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.773028 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-envoy" pod "95aff2e6-4643-40f2-a714-46ec991854a4" (UID: "95aff2e6-4643-40f2-a714-46ec991854a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:45.872757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.872676 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aff2e6-4643-40f2-a714-46ec991854a4-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.872757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.872707 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.872757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.872717 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aff2e6-4643-40f2-a714-46ec991854a4-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.872757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.872727 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvgg6\" (UniqueName: \"kubernetes.io/projected/95aff2e6-4643-40f2-a714-46ec991854a4-kube-api-access-qvgg6\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.872757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.872739 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aff2e6-4643-40f2-a714-46ec991854a4-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:45.991352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.991311 4898 generic.go:356] "Generic (PLEG): container finished" podID="95aff2e6-4643-40f2-a714-46ec991854a4" containerID="be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02" exitCode=0 Nov 03 22:57:45.991501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.991358 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" Nov 03 22:57:45.991501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.991376 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" event={"ID":"95aff2e6-4643-40f2-a714-46ec991854a4","Type":"ContainerDied","Data":"be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02"} Nov 03 22:57:45.991501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.991427 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz" event={"ID":"95aff2e6-4643-40f2-a714-46ec991854a4","Type":"ContainerDied","Data":"c068fec52269b5b0d132a944dfb4aa3f0ce5d276afc3f34b6d58c0137874249d"} Nov 03 22:57:45.991501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:45.991455 4898 scope.go:122] "RemoveContainer" containerID="be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02" Nov 03 22:57:46.003345 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:46.003325 4898 scope.go:122] "RemoveContainer" containerID="be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02" Nov 03 22:57:46.003687 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:57:46.003664 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02\": container with ID starting with be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02 not found: ID does not exist" containerID="be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02" Nov 03 22:57:46.003815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:46.003693 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02"} err="failed to get container status \"be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02\": rpc error: code = NotFound desc = could not find container \"be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02\": container with ID starting with be3ad1cb4df2de7e6054eaa88a668c0fd0059920103e10bb7a9b79b9eadf3f02 not found: ID does not exist" Nov 03 22:57:46.031125 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:46.031089 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:57:46.031268 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:46.031139 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 22:57:46.163699 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:46.163599 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz"] Nov 03 22:57:46.166522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:46.166496 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-kp9cz"] Nov 03 22:57:46.907393 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:46.907315 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" path="/var/lib/kubelet/pods/95aff2e6-4643-40f2-a714-46ec991854a4/volumes" Nov 03 22:57:47.031634 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:47.031598 4898 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:57:47.031857 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:47.031667 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 22:57:48.036927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:48.036895 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:48.037425 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:48.037271 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:48.038083 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:57:48.038065 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 22:57:49.836542 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:57:49.836503 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:58:02.452391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:02.452327 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4"] Nov 03 22:58:02.454087 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:02.454027 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" podUID="1294603d-a69d-47a6-9a02-42051d7afefe" containerName="maas-api" containerID="cri-o://3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4" gracePeriod=30 Nov 03 22:58:02.951571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:02.940540 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:58:03.007979 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.007905 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-metrics-certs\") pod \"1294603d-a69d-47a6-9a02-42051d7afefe\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " Nov 03 22:58:03.008192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.008178 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1294603d-a69d-47a6-9a02-42051d7afefe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1294603d-a69d-47a6-9a02-42051d7afefe-openshift-service-ca\") pod \"1294603d-a69d-47a6-9a02-42051d7afefe\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " Nov 03 22:58:03.008305 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.008293 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls\") pod \"1294603d-a69d-47a6-9a02-42051d7afefe\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " Nov 03 22:58:03.008403 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.008390 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1294603d-a69d-47a6-9a02-42051d7afefe-kube-api-access-69wd2\" (UniqueName: \"kubernetes.io/projected/1294603d-a69d-47a6-9a02-42051d7afefe-kube-api-access-69wd2\") pod \"1294603d-a69d-47a6-9a02-42051d7afefe\" (UID: \"1294603d-a69d-47a6-9a02-42051d7afefe\") " Nov 03 22:58:03.009369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.009343 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1294603d-a69d-47a6-9a02-42051d7afefe-openshift-service-ca" pod "1294603d-a69d-47a6-9a02-42051d7afefe" (UID: "1294603d-a69d-47a6-9a02-42051d7afefe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:03.012369 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.012333 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1294603d-a69d-47a6-9a02-42051d7afefe-kube-api-access-69wd2" pod "1294603d-a69d-47a6-9a02-42051d7afefe" (UID: "1294603d-a69d-47a6-9a02-42051d7afefe"). InnerVolumeSpecName "kube-api-access-69wd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:03.012845 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.012814 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls" pod "1294603d-a69d-47a6-9a02-42051d7afefe" (UID: "1294603d-a69d-47a6-9a02-42051d7afefe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:03.027076 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.026905 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-metrics-certs" pod "1294603d-a69d-47a6-9a02-42051d7afefe" (UID: "1294603d-a69d-47a6-9a02-42051d7afefe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:03.110097 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.110060 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:03.110265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.110101 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1294603d-a69d-47a6-9a02-42051d7afefe-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:03.110265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.110150 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1294603d-a69d-47a6-9a02-42051d7afefe-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:03.110265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.110166 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69wd2\" (UniqueName: \"kubernetes.io/projected/1294603d-a69d-47a6-9a02-42051d7afefe-kube-api-access-69wd2\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:03.118475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.118439 4898 generic.go:356] "Generic (PLEG): container finished" podID="1294603d-a69d-47a6-9a02-42051d7afefe" containerID="3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4" exitCode=0 Nov 03 22:58:03.118615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.118498 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" event={"ID":"1294603d-a69d-47a6-9a02-42051d7afefe","Type":"ContainerDied","Data":"3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4"} Nov 03 22:58:03.118615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.118524 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" Nov 03 22:58:03.118615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.118552 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4" event={"ID":"1294603d-a69d-47a6-9a02-42051d7afefe","Type":"ContainerDied","Data":"5820bd520ac8b25fc237f4205f95a9c11884fb61bebffbe274f99952e3ad9202"} Nov 03 22:58:03.118615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.118582 4898 scope.go:122] "RemoveContainer" containerID="3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4" Nov 03 22:58:03.133239 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.133026 4898 scope.go:122] "RemoveContainer" containerID="3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4" Nov 03 22:58:03.133422 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:03.133396 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4\": container with ID starting with 3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4 not found: ID does not exist" containerID="3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4" Nov 03 22:58:03.133477 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.133436 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4"} err="failed to get container status \"3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4\": rpc error: code = NotFound desc = could not find container \"3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4\": container with ID starting with 3c387c55d2cb1f6670a05c4b5931d2fae66bef44f488d94e86b35025ef137fd4 not found: ID does not exist" Nov 03 22:58:03.285378 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.285303 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4"] Nov 03 22:58:03.289092 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:03.289064 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-65913e-7dc6fb8564-x88n4"] Nov 03 22:58:04.101515 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.101482 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq"] Nov 03 22:58:04.102259 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.102238 4898 cpu_manager.go:414] "containerMap: removing container" podUID="1294603d-a69d-47a6-9a02-42051d7afefe" containerName="maas-api" Nov 03 22:58:04.102310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.102262 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1294603d-a69d-47a6-9a02-42051d7afefe" containerName="maas-api" Nov 03 22:58:04.102310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.102277 4898 cpu_manager.go:414] "containerMap: removing container" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" Nov 03 22:58:04.102310 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.102282 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" Nov 03 22:58:04.102408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.102395 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1294603d-a69d-47a6-9a02-42051d7afefe" containerName="maas-api" Nov 03 22:58:04.102443 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.102415 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aff2e6-4643-40f2-a714-46ec991854a4" containerName="istio-proxy" Nov 03 22:58:04.107399 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.107381 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.121484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.121458 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.121616 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.121597 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xm4\" (UniqueName: \"kubernetes.io/projected/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-kube-api-access-l8xm4\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.121721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.121705 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.121779 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.121764 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-metrics-certs\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.163547 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.163507 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq"] Nov 03 22:58:04.208803 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.208762 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9b4383-serving-cert\"" Nov 03 22:58:04.222763 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.222729 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xm4\" (UniqueName: \"kubernetes.io/projected/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-kube-api-access-l8xm4\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.222952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.222832 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.222952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.222884 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-metrics-certs\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.222952 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.222935 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.223119 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:04.222977 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-serving-cert: secret "maas-api-e2e-inf-a-9b4383-serving-cert" not found Nov 03 22:58:04.223119 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:04.223040 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls podName:8d717beb-5c93-4f7c-b3dc-f925dd9ee844 nodeName:}" failed. No retries permitted until 2026-11-03 22:58:04.723019246 +0000 UTC m=+2127.935788292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls") pod "maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" (UID: "8d717beb-5c93-4f7c-b3dc-f925dd9ee844") : secret "maas-api-e2e-inf-a-9b4383-serving-cert" not found Nov 03 22:58:04.223720 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.223695 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.225537 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.225512 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-metrics-certs\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.286608 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.286574 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xm4\" (UniqueName: \"kubernetes.io/projected/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-kube-api-access-l8xm4\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.730174 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.730136 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.732526 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.732498 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:04.906727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:04.906695 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1294603d-a69d-47a6-9a02-42051d7afefe" path="/var/lib/kubelet/pods/1294603d-a69d-47a6-9a02-42051d7afefe/volumes" Nov 03 22:58:05.021361 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:05.021260 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:05.400750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:05.400720 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq"] Nov 03 22:58:05.401697 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:58:05.401671 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d717beb_5c93_4f7c_b3dc_f925dd9ee844.slice/crio-66815e818ef413dc227c8f3f7b8098dcbf64835d15797f04007b6426fdb4fb32 WatchSource:0}: Error finding container 66815e818ef413dc227c8f3f7b8098dcbf64835d15797f04007b6426fdb4fb32: Status 404 returned error can't find the container with id 66815e818ef413dc227c8f3f7b8098dcbf64835d15797f04007b6426fdb4fb32 Nov 03 22:58:06.148255 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:06.148214 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" event={"ID":"8d717beb-5c93-4f7c-b3dc-f925dd9ee844","Type":"ContainerStarted","Data":"66815e818ef413dc227c8f3f7b8098dcbf64835d15797f04007b6426fdb4fb32"} Nov 03 22:58:07.158966 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:07.158916 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" event={"ID":"8d717beb-5c93-4f7c-b3dc-f925dd9ee844","Type":"ContainerStarted","Data":"14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad"} Nov 03 22:58:07.159349 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:07.159002 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:07.278652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:07.278597 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" podStartSLOduration=2.432631708 podStartE2EDuration="3.278581995s" totalImagesPullingTime="845.950287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:07.277427979 +0000 UTC m=+2130.490197030" watchObservedRunningTime="2026-11-03 22:58:07.278581995 +0000 UTC m=+2130.491351047" Nov 03 22:58:08.483063 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.483029 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7243-8da5f0"] Nov 03 22:58:08.487382 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.487364 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:08.538991 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.538958 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-8da5f0"] Nov 03 22:58:08.584735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.584695 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j4jqv\"" Nov 03 22:58:08.586478 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.586444 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmgw\" (UniqueName: \"kubernetes.io/projected/a3114bdf-d4c3-4e0f-870a-645040561114-kube-api-access-qdmgw\") pod \"test-curl-7243-8da5f0\" (UID: \"a3114bdf-d4c3-4e0f-870a-645040561114\") " pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:08.688222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.688181 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmgw\" (UniqueName: \"kubernetes.io/projected/a3114bdf-d4c3-4e0f-870a-645040561114-kube-api-access-qdmgw\") pod \"test-curl-7243-8da5f0\" (UID: \"a3114bdf-d4c3-4e0f-870a-645040561114\") " pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:08.749434 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.749356 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmgw\" (UniqueName: \"kubernetes.io/projected/a3114bdf-d4c3-4e0f-870a-645040561114-kube-api-access-qdmgw\") pod \"test-curl-7243-8da5f0\" (UID: \"a3114bdf-d4c3-4e0f-870a-645040561114\") " pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:08.800625 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:08.800588 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:09.182082 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:09.182054 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-8da5f0"] Nov 03 22:58:09.183888 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:58:09.183854 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3114bdf_d4c3_4e0f_870a_645040561114.slice/crio-0e5b4aec76fb78bf8c83d2ce254f3486716eb1abe20e5e0312a4e46005dcfa9c WatchSource:0}: Error finding container 0e5b4aec76fb78bf8c83d2ce254f3486716eb1abe20e5e0312a4e46005dcfa9c: Status 404 returned error can't find the container with id 0e5b4aec76fb78bf8c83d2ce254f3486716eb1abe20e5e0312a4e46005dcfa9c Nov 03 22:58:10.179855 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:10.179812 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-8da5f0" event={"ID":"a3114bdf-d4c3-4e0f-870a-645040561114","Type":"ContainerStarted","Data":"2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221"} Nov 03 22:58:10.180277 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:10.179866 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-8da5f0" event={"ID":"a3114bdf-d4c3-4e0f-870a-645040561114","Type":"ContainerStarted","Data":"0e5b4aec76fb78bf8c83d2ce254f3486716eb1abe20e5e0312a4e46005dcfa9c"} Nov 03 22:58:10.294922 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:10.294867 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7243-8da5f0" podStartSLOduration=2.294851049 podStartE2EDuration="2.294851049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:10.293357288 +0000 UTC m=+2133.506126341" watchObservedRunningTime="2026-11-03 22:58:10.294851049 +0000 UTC m=+2133.507620099" Nov 03 22:58:11.633552 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:11.633518 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk"] Nov 03 22:58:11.634038 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:11.633759 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" containerID="cri-o://7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4" gracePeriod=30 Nov 03 22:58:11.773353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:11.773320 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7243-8da5f0"] Nov 03 22:58:11.785247 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:11.785193 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7243-8da5f0"] Nov 03 22:58:12.192960 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:12.192896 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7243-8da5f0" podUID="a3114bdf-d4c3-4e0f-870a-645040561114" containerName="test-curl-7243-8da5f0" containerID="cri-o://2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221" gracePeriod=2 Nov 03 22:58:12.244709 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:12.244673 4898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7243-8da5f0\" is forbidden: User \"system:node:ip-10-0-139-173.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-173.us-west-2.compute.internal' and this object" podUID="a3114bdf-d4c3-4e0f-870a-645040561114" pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:13.171941 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:13.171913 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 22:58:14.008522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:14.008482 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:58:14.008724 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:14.008540 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:58:14.464342 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:14.464309 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:14.576673 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:14.576642 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3114bdf-d4c3-4e0f-870a-645040561114-kube-api-access-qdmgw\" (UniqueName: \"kubernetes.io/projected/a3114bdf-d4c3-4e0f-870a-645040561114-kube-api-access-qdmgw\") pod \"a3114bdf-d4c3-4e0f-870a-645040561114\" (UID: \"a3114bdf-d4c3-4e0f-870a-645040561114\") " Nov 03 22:58:14.578854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:14.578819 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3114bdf-d4c3-4e0f-870a-645040561114-kube-api-access-qdmgw" pod "a3114bdf-d4c3-4e0f-870a-645040561114" (UID: "a3114bdf-d4c3-4e0f-870a-645040561114"). InnerVolumeSpecName "kube-api-access-qdmgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:14.678496 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:14.678398 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdmgw\" (UniqueName: \"kubernetes.io/projected/a3114bdf-d4c3-4e0f-870a-645040561114-kube-api-access-qdmgw\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:14.908188 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:14.908151 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3114bdf-d4c3-4e0f-870a-645040561114" path="/var/lib/kubelet/pods/a3114bdf-d4c3-4e0f-870a-645040561114/volumes" Nov 03 22:58:15.215155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:15.215119 4898 generic.go:356] "Generic (PLEG): container finished" podID="a3114bdf-d4c3-4e0f-870a-645040561114" containerID="2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221" exitCode=137 Nov 03 22:58:15.215365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:15.215161 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-8da5f0" Nov 03 22:58:15.215365 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:15.215218 4898 scope.go:122] "RemoveContainer" containerID="2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221" Nov 03 22:58:15.224992 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:15.224973 4898 scope.go:122] "RemoveContainer" containerID="2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221" Nov 03 22:58:15.225267 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:15.225249 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221\": container with ID starting with 2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221 not found: ID does not exist" containerID="2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221" Nov 03 22:58:15.225331 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:15.225281 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221"} err="failed to get container status \"2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221\": rpc error: code = NotFound desc = could not find container \"2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221\": container with ID starting with 2c653adc6d7107d52124482418d9cb4c6734c8a21cc4c95e8399e6da94aad221 not found: ID does not exist" Nov 03 22:58:16.910323 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:16.910262 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:58:17.007686 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.007652 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-kube-api-access-gl4t4\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-kube-api-access-gl4t4\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.007927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.007705 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-envoy\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.007927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.007748 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-podinfo\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.007927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.007821 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-certs\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.007927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.007887 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-data\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.008159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.007978 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-socket\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.008159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008052 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-credential-socket\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.008159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008086 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-token\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-token\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.008159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008147 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952a43e1-5149-45e3-8c2e-55eec1b64d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952a43e1-5149-45e3-8c2e-55eec1b64d63-istiod-ca-cert\") pod \"952a43e1-5149-45e3-8c2e-55eec1b64d63\" (UID: \"952a43e1-5149-45e3-8c2e-55eec1b64d63\") " Nov 03 22:58:17.008368 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008185 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-certs" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:17.008368 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008242 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-data" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:17.008476 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008427 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-socket" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:17.008581 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008517 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-credential-socket" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:17.008747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008720 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952a43e1-5149-45e3-8c2e-55eec1b64d63-istiod-ca-cert" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:17.008865 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008845 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.008936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008872 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.008936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008888 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.008936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008902 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.008936 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.008917 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952a43e1-5149-45e3-8c2e-55eec1b64d63-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.010256 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.010189 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-envoy" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:17.010256 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.010241 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-podinfo" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:17.010375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.010316 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-token" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:17.010375 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.010344 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-kube-api-access-gl4t4" pod "952a43e1-5149-45e3-8c2e-55eec1b64d63" (UID: "952a43e1-5149-45e3-8c2e-55eec1b64d63"). InnerVolumeSpecName "kube-api-access-gl4t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:17.110475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.110434 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.110475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.110463 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl4t4\" (UniqueName: \"kubernetes.io/projected/952a43e1-5149-45e3-8c2e-55eec1b64d63-kube-api-access-gl4t4\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.110475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.110473 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.110475 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.110484 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952a43e1-5149-45e3-8c2e-55eec1b64d63-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.231905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.231809 4898 generic.go:356] "Generic (PLEG): container finished" podID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerID="7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4" exitCode=0 Nov 03 22:58:17.231905 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.231865 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" Nov 03 22:58:17.232121 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.231917 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" event={"ID":"952a43e1-5149-45e3-8c2e-55eec1b64d63","Type":"ContainerDied","Data":"7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4"} Nov 03 22:58:17.232121 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.231981 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk" event={"ID":"952a43e1-5149-45e3-8c2e-55eec1b64d63","Type":"ContainerDied","Data":"9538b5e53dbcb454a415deef4ad1f618a8e96a2481df698cc5daf8903fb781ed"} Nov 03 22:58:17.232121 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.232014 4898 scope.go:122] "RemoveContainer" containerID="7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4" Nov 03 22:58:17.246141 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.246119 4898 scope.go:122] "RemoveContainer" containerID="b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd" Nov 03 22:58:17.258159 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.258135 4898 scope.go:122] "RemoveContainer" containerID="7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4" Nov 03 22:58:17.258498 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:17.258473 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4\": container with ID starting with 7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4 not found: ID does not exist" containerID="7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4" Nov 03 22:58:17.258570 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.258505 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4"} err="failed to get container status \"7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4\": rpc error: code = NotFound desc = could not find container \"7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4\": container with ID starting with 7d94c6a7c7e11a9b0c061737396250d5afb817a0ad23e6ae615ebc332e2a8ff4 not found: ID does not exist" Nov 03 22:58:17.258570 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.258526 4898 scope.go:122] "RemoveContainer" containerID="b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd" Nov 03 22:58:17.258802 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:17.258769 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd\": container with ID starting with b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd not found: ID does not exist" containerID="b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd" Nov 03 22:58:17.258886 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.258867 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd"} err="failed to get container status \"b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd\": rpc error: code = NotFound desc = could not find container \"b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd\": container with ID starting with b10567a6a0efa8fd6fabab5fabf0fab29c1367e99088cd1e5815bfb0dcc935cd not found: ID does not exist" Nov 03 22:58:17.399754 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.399720 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk"] Nov 03 22:58:17.403465 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:17.403440 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-65913e-openshift-default-65986b75f9-s5cnk"] Nov 03 22:58:18.909252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:18.909218 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" path="/var/lib/kubelet/pods/952a43e1-5149-45e3-8c2e-55eec1b64d63/volumes" Nov 03 22:58:19.372927 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.372889 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg"] Nov 03 22:58:19.373263 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.373140 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" podUID="8c88f565-1f64-4542-9a7a-b2390803b9d7" containerName="maas-api" containerID="cri-o://e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c" gracePeriod=30 Nov 03 22:58:19.710877 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.710850 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:58:19.867815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.864973 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls\") pod \"8c88f565-1f64-4542-9a7a-b2390803b9d7\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " Nov 03 22:58:19.867815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.865033 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c88f565-1f64-4542-9a7a-b2390803b9d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c88f565-1f64-4542-9a7a-b2390803b9d7-openshift-service-ca\") pod \"8c88f565-1f64-4542-9a7a-b2390803b9d7\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " Nov 03 22:58:19.867815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.865150 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-metrics-certs\") pod \"8c88f565-1f64-4542-9a7a-b2390803b9d7\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " Nov 03 22:58:19.867815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.865201 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c88f565-1f64-4542-9a7a-b2390803b9d7-kube-api-access-99nlv\" (UniqueName: \"kubernetes.io/projected/8c88f565-1f64-4542-9a7a-b2390803b9d7-kube-api-access-99nlv\") pod \"8c88f565-1f64-4542-9a7a-b2390803b9d7\" (UID: \"8c88f565-1f64-4542-9a7a-b2390803b9d7\") " Nov 03 22:58:19.875663 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.875125 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c88f565-1f64-4542-9a7a-b2390803b9d7-openshift-service-ca" pod "8c88f565-1f64-4542-9a7a-b2390803b9d7" (UID: "8c88f565-1f64-4542-9a7a-b2390803b9d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:19.891831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.881340 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-metrics-certs" pod "8c88f565-1f64-4542-9a7a-b2390803b9d7" (UID: "8c88f565-1f64-4542-9a7a-b2390803b9d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:19.891831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.881639 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c88f565-1f64-4542-9a7a-b2390803b9d7-kube-api-access-99nlv" pod "8c88f565-1f64-4542-9a7a-b2390803b9d7" (UID: "8c88f565-1f64-4542-9a7a-b2390803b9d7"). InnerVolumeSpecName "kube-api-access-99nlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:19.891831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.882021 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls" pod "8c88f565-1f64-4542-9a7a-b2390803b9d7" (UID: "8c88f565-1f64-4542-9a7a-b2390803b9d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:19.897422 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:19.897170 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:58:19.966696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.966598 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:19.966696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.966638 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c88f565-1f64-4542-9a7a-b2390803b9d7-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:19.966696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.966654 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88f565-1f64-4542-9a7a-b2390803b9d7-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:19.966696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:19.966669 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99nlv\" (UniqueName: \"kubernetes.io/projected/8c88f565-1f64-4542-9a7a-b2390803b9d7-kube-api-access-99nlv\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:20.257566 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.257466 4898 generic.go:356] "Generic (PLEG): container finished" podID="8c88f565-1f64-4542-9a7a-b2390803b9d7" containerID="e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c" exitCode=0 Nov 03 22:58:20.257566 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.257519 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" Nov 03 22:58:20.257566 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.257539 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" event={"ID":"8c88f565-1f64-4542-9a7a-b2390803b9d7","Type":"ContainerDied","Data":"e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c"} Nov 03 22:58:20.257878 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.257611 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg" event={"ID":"8c88f565-1f64-4542-9a7a-b2390803b9d7","Type":"ContainerDied","Data":"86796d8af242d9e0d301cd805d85fc8b2e77324a80ccc45b32befcc999fbb422"} Nov 03 22:58:20.257878 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.257640 4898 scope.go:122] "RemoveContainer" containerID="e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c" Nov 03 22:58:20.270902 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.270872 4898 scope.go:122] "RemoveContainer" containerID="e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c" Nov 03 22:58:20.271223 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:20.271202 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c\": container with ID starting with e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c not found: ID does not exist" containerID="e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c" Nov 03 22:58:20.271315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.271233 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c"} err="failed to get container status \"e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c\": rpc error: code = NotFound desc = could not find container \"e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c\": container with ID starting with e9bf0650de5fb9a3bec20a040825028ff6bd68137a8f4d44c635e21c55e8475c not found: ID does not exist" Nov 03 22:58:20.424137 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.424105 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg"] Nov 03 22:58:20.429735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.429706 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0189ef-6f4985c5fb-rrkhg"] Nov 03 22:58:20.908080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:20.908047 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c88f565-1f64-4542-9a7a-b2390803b9d7" path="/var/lib/kubelet/pods/8c88f565-1f64-4542-9a7a-b2390803b9d7/volumes" Nov 03 22:58:29.049671 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:29.049625 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f"] Nov 03 22:58:29.050192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:29.049890 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" containerID="cri-o://ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c" gracePeriod=30 Nov 03 22:58:30.282967 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:30.282929 4898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:58:30.283457 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:30.282996 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:58:30.813891 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:30.813840 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl"] Nov 03 22:58:30.814124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:30.814057 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" podUID="220db423-ef85-4549-a07f-353c37c06192" containerName="maas-api" containerID="cri-o://69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659" gracePeriod=30 Nov 03 22:58:31.181607 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.181571 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:58:31.254811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.252689 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-metrics-certs\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-metrics-certs\") pod \"220db423-ef85-4549-a07f-353c37c06192\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " Nov 03 22:58:31.254811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.252965 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls\") pod \"220db423-ef85-4549-a07f-353c37c06192\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " Nov 03 22:58:31.254811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.253015 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220db423-ef85-4549-a07f-353c37c06192-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220db423-ef85-4549-a07f-353c37c06192-openshift-service-ca\") pod \"220db423-ef85-4549-a07f-353c37c06192\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " Nov 03 22:58:31.254811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.253115 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220db423-ef85-4549-a07f-353c37c06192-kube-api-access-grwvk\" (UniqueName: \"kubernetes.io/projected/220db423-ef85-4549-a07f-353c37c06192-kube-api-access-grwvk\") pod \"220db423-ef85-4549-a07f-353c37c06192\" (UID: \"220db423-ef85-4549-a07f-353c37c06192\") " Nov 03 22:58:31.254811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.254776 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220db423-ef85-4549-a07f-353c37c06192-openshift-service-ca" pod "220db423-ef85-4549-a07f-353c37c06192" (UID: "220db423-ef85-4549-a07f-353c37c06192"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:31.302522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.292960 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-metrics-certs" pod "220db423-ef85-4549-a07f-353c37c06192" (UID: "220db423-ef85-4549-a07f-353c37c06192"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:31.302522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.293096 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls" pod "220db423-ef85-4549-a07f-353c37c06192" (UID: "220db423-ef85-4549-a07f-353c37c06192"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:31.302522 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.293175 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220db423-ef85-4549-a07f-353c37c06192-kube-api-access-grwvk" pod "220db423-ef85-4549-a07f-353c37c06192" (UID: "220db423-ef85-4549-a07f-353c37c06192"). InnerVolumeSpecName "kube-api-access-grwvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:31.342900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.339855 4898 generic.go:356] "Generic (PLEG): container finished" podID="220db423-ef85-4549-a07f-353c37c06192" containerID="69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659" exitCode=0 Nov 03 22:58:31.342900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.340024 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" event={"ID":"220db423-ef85-4549-a07f-353c37c06192","Type":"ContainerDied","Data":"69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659"} Nov 03 22:58:31.342900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.340095 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" event={"ID":"220db423-ef85-4549-a07f-353c37c06192","Type":"ContainerDied","Data":"b80d1600bfaaa1d3b5b48d86d48f490b8db6734a1d3858120ce6e3083134368e"} Nov 03 22:58:31.342900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.340140 4898 scope.go:122] "RemoveContainer" containerID="69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659" Nov 03 22:58:31.342900 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.340333 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl" Nov 03 22:58:31.355073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.355041 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:31.355073 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.355076 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220db423-ef85-4549-a07f-353c37c06192-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:31.355232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.355092 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220db423-ef85-4549-a07f-353c37c06192-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:31.355232 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.355108 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grwvk\" (UniqueName: \"kubernetes.io/projected/220db423-ef85-4549-a07f-353c37c06192-kube-api-access-grwvk\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:31.368140 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.368116 4898 scope.go:122] "RemoveContainer" containerID="69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659" Nov 03 22:58:31.368511 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:31.368491 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659\": container with ID starting with 69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659 not found: ID does not exist" containerID="69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659" Nov 03 22:58:31.368611 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.368522 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659"} err="failed to get container status \"69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659\": rpc error: code = NotFound desc = could not find container \"69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659\": container with ID starting with 69a2695e0a1e943a0597bb88e3937ff3a65cdd58e940755d30befa45acf91659 not found: ID does not exist" Nov 03 22:58:31.512680 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.512589 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl"] Nov 03 22:58:31.516898 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:31.516872 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c21c7-f5569d595-z59wl"] Nov 03 22:58:32.912501 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:32.912465 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220db423-ef85-4549-a07f-353c37c06192" path="/var/lib/kubelet/pods/220db423-ef85-4549-a07f-353c37c06192/volumes" Nov 03 22:58:34.364322 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.364293 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:58:34.364781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.364398 4898 generic.go:356] "Generic (PLEG): container finished" podID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerID="ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c" exitCode=0 Nov 03 22:58:34.364781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.364459 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" event={"ID":"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3","Type":"ContainerDied","Data":"ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c"} Nov 03 22:58:34.364781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.364503 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" event={"ID":"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3","Type":"ContainerDied","Data":"ed6d93e4c3e0a566dff205bf0a991c8405245302c7a6faa4eb3f0f24dca343ac"} Nov 03 22:58:34.364781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.364527 4898 scope.go:122] "RemoveContainer" containerID="ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c" Nov 03 22:58:34.377069 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.377047 4898 scope.go:122] "RemoveContainer" containerID="81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274" Nov 03 22:58:34.386429 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.386410 4898 scope.go:122] "RemoveContainer" containerID="ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c" Nov 03 22:58:34.386754 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:34.386728 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c\": container with ID starting with ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c not found: ID does not exist" containerID="ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c" Nov 03 22:58:34.386871 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.386765 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c"} err="failed to get container status \"ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c\": rpc error: code = NotFound desc = could not find container \"ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c\": container with ID starting with ffa96c42ab6be08acf494fc71cda5025953659103a92c033763dfc42d8715b0c not found: ID does not exist" Nov 03 22:58:34.386871 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.386784 4898 scope.go:122] "RemoveContainer" containerID="81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274" Nov 03 22:58:34.387077 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:34.387056 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274\": container with ID starting with 81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274 not found: ID does not exist" containerID="81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274" Nov 03 22:58:34.387119 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.387089 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274"} err="failed to get container status \"81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274\": rpc error: code = NotFound desc = could not find container \"81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274\": container with ID starting with 81c0ffd0300c6bb2745161668ce2790d01e1f2567515046f1c08e9f368f1e274 not found: ID does not exist" Nov 03 22:58:34.499461 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499426 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-podinfo\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.499647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499491 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-socket\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.499647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499581 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istiod-ca-cert\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.499749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499669 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-envoy\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.499749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499716 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-certs\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.499889 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499824 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-socket" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:34.499958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499901 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-credential-socket\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.500016 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499990 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-kube-api-access-5r8cl\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-kube-api-access-5r8cl\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.500099 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.499942 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istiod-ca-cert" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:34.500157 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500097 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-data\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.500157 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500004 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-certs" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:34.500157 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500149 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-token\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-token\") pod \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\" (UID: \"cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3\") " Nov 03 22:58:34.500157 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500108 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-credential-socket" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:34.500340 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500281 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-data" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:34.500770 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500748 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.500770 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500768 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.500980 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500777 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.500980 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500816 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.500980 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.500826 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.502254 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.502225 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-token" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:34.502402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.502369 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-kube-api-access-5r8cl" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "kube-api-access-5r8cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:34.502511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.502417 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-envoy" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:34.502511 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.502460 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-podinfo" pod "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" (UID: "cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:34.602640 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.602565 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.602640 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.602592 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.602640 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.602602 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r8cl\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-kube-api-access-5r8cl\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:34.602640 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:34.602612 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:35.372552 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:35.372515 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f" Nov 03 22:58:35.589764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:35.589731 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f"] Nov 03 22:58:35.640052 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:35.639967 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0189ef-openshift-default-cd86cdd47-8h87f"] Nov 03 22:58:36.907018 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:36.906985 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" path="/var/lib/kubelet/pods/cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3/volumes" Nov 03 22:58:37.304255 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.304179 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc"] Nov 03 22:58:37.304404 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.304378 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" podUID="c0f3f17a-a76b-47cb-910b-706b8f01bbb1" containerName="maas-api" containerID="cri-o://3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681" gracePeriod=30 Nov 03 22:58:37.626925 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.626893 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:58:37.740984 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.740943 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-metrics-certs\") pod \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " Nov 03 22:58:37.741170 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.741014 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls\") pod \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " Nov 03 22:58:37.741250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.741179 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-openshift-service-ca\") pod \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " Nov 03 22:58:37.741250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.741235 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-kube-api-access-52q2w\" (UniqueName: \"kubernetes.io/projected/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-kube-api-access-52q2w\") pod \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\" (UID: \"c0f3f17a-a76b-47cb-910b-706b8f01bbb1\") " Nov 03 22:58:37.742039 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.741700 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-openshift-service-ca" pod "c0f3f17a-a76b-47cb-910b-706b8f01bbb1" (UID: "c0f3f17a-a76b-47cb-910b-706b8f01bbb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:37.742039 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.741995 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:37.743707 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.743682 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-kube-api-access-52q2w" pod "c0f3f17a-a76b-47cb-910b-706b8f01bbb1" (UID: "c0f3f17a-a76b-47cb-910b-706b8f01bbb1"). InnerVolumeSpecName "kube-api-access-52q2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:37.743814 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.743765 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls" pod "c0f3f17a-a76b-47cb-910b-706b8f01bbb1" (UID: "c0f3f17a-a76b-47cb-910b-706b8f01bbb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:37.747260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.747233 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-metrics-certs" pod "c0f3f17a-a76b-47cb-910b-706b8f01bbb1" (UID: "c0f3f17a-a76b-47cb-910b-706b8f01bbb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:37.846458 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.846369 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:37.846652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.846634 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:37.846804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:37.846745 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52q2w\" (UniqueName: \"kubernetes.io/projected/c0f3f17a-a76b-47cb-910b-706b8f01bbb1-kube-api-access-52q2w\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:38.405380 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.405342 4898 generic.go:356] "Generic (PLEG): container finished" podID="c0f3f17a-a76b-47cb-910b-706b8f01bbb1" containerID="3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681" exitCode=0 Nov 03 22:58:38.405973 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.405412 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" Nov 03 22:58:38.406854 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.405435 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" event={"ID":"c0f3f17a-a76b-47cb-910b-706b8f01bbb1","Type":"ContainerDied","Data":"3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681"} Nov 03 22:58:38.407012 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.406977 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc" event={"ID":"c0f3f17a-a76b-47cb-910b-706b8f01bbb1","Type":"ContainerDied","Data":"7ef4116f07f5349dbad7c1a288cc1b92728764b37c41136ba58a2483e70db71b"} Nov 03 22:58:38.407092 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.407024 4898 scope.go:122] "RemoveContainer" containerID="3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681" Nov 03 22:58:38.424637 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.424612 4898 scope.go:122] "RemoveContainer" containerID="3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681" Nov 03 22:58:38.424985 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:38.424963 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681\": container with ID starting with 3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681 not found: ID does not exist" containerID="3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681" Nov 03 22:58:38.425058 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.424994 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681"} err="failed to get container status \"3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681\": rpc error: code = NotFound desc = could not find container \"3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681\": container with ID starting with 3c7cc1777a8ac3ea01c959599a3d203b2888aac144c65cd10757e09f3b0a3681 not found: ID does not exist" Nov 03 22:58:38.579088 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.579053 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc"] Nov 03 22:58:38.628265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.628226 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2420f6-6b87d57c6-p96vc"] Nov 03 22:58:38.911347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:38.911302 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f3f17a-a76b-47cb-910b-706b8f01bbb1" path="/var/lib/kubelet/pods/c0f3f17a-a76b-47cb-910b-706b8f01bbb1/volumes" Nov 03 22:58:39.129612 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.129580 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz"] Nov 03 22:58:39.130240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130223 4898 cpu_manager.go:414] "containerMap: removing container" podUID="220db423-ef85-4549-a07f-353c37c06192" containerName="maas-api" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130241 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220db423-ef85-4549-a07f-353c37c06192" containerName="maas-api" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130254 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a3114bdf-d4c3-4e0f-870a-645040561114" containerName="test-curl-7243-8da5f0" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130261 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3114bdf-d4c3-4e0f-870a-645040561114" containerName="test-curl-7243-8da5f0" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130272 4898 cpu_manager.go:414] "containerMap: removing container" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130277 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130296 4898 cpu_manager.go:414] "containerMap: removing container" podUID="8c88f565-1f64-4542-9a7a-b2390803b9d7" containerName="maas-api" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130301 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c88f565-1f64-4542-9a7a-b2390803b9d7" containerName="maas-api" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130310 4898 cpu_manager.go:414] "containerMap: removing container" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" Nov 03 22:58:39.130315 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130314 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130325 4898 cpu_manager.go:414] "containerMap: removing container" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130330 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130340 4898 cpu_manager.go:414] "containerMap: removing container" podUID="c0f3f17a-a76b-47cb-910b-706b8f01bbb1" containerName="maas-api" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130345 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f3f17a-a76b-47cb-910b-706b8f01bbb1" containerName="maas-api" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130355 4898 cpu_manager.go:414] "containerMap: removing container" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130360 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130450 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c88f565-1f64-4542-9a7a-b2390803b9d7" containerName="maas-api" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130460 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130467 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130475 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f3f17a-a76b-47cb-910b-706b8f01bbb1" containerName="maas-api" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130480 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="952a43e1-5149-45e3-8c2e-55eec1b64d63" containerName="istio-proxy" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130489 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="220db423-ef85-4549-a07f-353c37c06192" containerName="maas-api" Nov 03 22:58:39.130591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.130494 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3114bdf-d4c3-4e0f-870a-645040561114" containerName="test-curl-7243-8da5f0" Nov 03 22:58:39.135308 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.135289 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.169173 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.169113 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0404aee4-fa58-4d0d-9eac-63ac403503d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.169286 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.169174 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4rl\" (UniqueName: \"kubernetes.io/projected/0404aee4-fa58-4d0d-9eac-63ac403503d5-kube-api-access-vw4rl\") pod \"payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.188997 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.188977 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-31ad7b-dockercfg-5mm22\"" Nov 03 22:58:39.191303 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.191279 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz"] Nov 03 22:58:39.262308 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.262277 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-31ad7b\"" Nov 03 22:58:39.294156 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.270743 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0404aee4-fa58-4d0d-9eac-63ac403503d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.294453 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.271606 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0404aee4-fa58-4d0d-9eac-63ac403503d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.295031 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.294919 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4rl\" (UniqueName: \"kubernetes.io/projected/0404aee4-fa58-4d0d-9eac-63ac403503d5-kube-api-access-vw4rl\") pod \"payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.298357 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.298331 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn"] Nov 03 22:58:39.299253 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.299232 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf648795-6e86-4ba4-95a8-a9aa0bd7d4c3" containerName="istio-proxy" Nov 03 22:58:39.305153 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.305135 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.346471 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.346441 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn"] Nov 03 22:58:39.360831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.358827 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4rl\" (UniqueName: \"kubernetes.io/projected/0404aee4-fa58-4d0d-9eac-63ac403503d5-kube-api-access-vw4rl\") pod \"payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.396418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.396387 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhnx\" (UniqueName: \"kubernetes.io/projected/06e66f7f-49d3-4e1d-9557-b8c57159c381-kube-api-access-tqhnx\") pod \"payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.396573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.396552 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e66f7f-49d3-4e1d-9557-b8c57159c381-plugins-config-volume\") pod \"payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.450573 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.450488 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:39.499755 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.499723 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e66f7f-49d3-4e1d-9557-b8c57159c381-plugins-config-volume\") pod \"payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.499974 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.499902 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhnx\" (UniqueName: \"kubernetes.io/projected/06e66f7f-49d3-4e1d-9557-b8c57159c381-kube-api-access-tqhnx\") pod \"payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.500918 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.500891 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e66f7f-49d3-4e1d-9557-b8c57159c381-plugins-config-volume\") pod \"payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.561656 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.561604 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhnx\" (UniqueName: \"kubernetes.io/projected/06e66f7f-49d3-4e1d-9557-b8c57159c381-kube-api-access-tqhnx\") pod \"payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.634258 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.634213 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:39.849160 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:39.849128 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz"] Nov 03 22:58:39.850324 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:58:39.850297 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0404aee4_fa58_4d0d_9eac_63ac403503d5.slice/crio-e4f52e9f637decc00b8fa48539f2db333759ff547006c6818ef895c715e4efeb WatchSource:0}: Error finding container e4f52e9f637decc00b8fa48539f2db333759ff547006c6818ef895c715e4efeb: Status 404 returned error can't find the container with id e4f52e9f637decc00b8fa48539f2db333759ff547006c6818ef895c715e4efeb Nov 03 22:58:40.021893 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.021862 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn"] Nov 03 22:58:40.023086 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:58:40.023062 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e66f7f_49d3_4e1d_9557_b8c57159c381.slice/crio-4ecc0a13522970682b4f60afa24502807bf66bce5949858ec7e03350175b8743 WatchSource:0}: Error finding container 4ecc0a13522970682b4f60afa24502807bf66bce5949858ec7e03350175b8743: Status 404 returned error can't find the container with id 4ecc0a13522970682b4f60afa24502807bf66bce5949858ec7e03350175b8743 Nov 03 22:58:40.421046 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.421004 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" event={"ID":"0404aee4-fa58-4d0d-9eac-63ac403503d5","Type":"ContainerStarted","Data":"96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928"} Nov 03 22:58:40.421260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.421064 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" event={"ID":"0404aee4-fa58-4d0d-9eac-63ac403503d5","Type":"ContainerStarted","Data":"e4f52e9f637decc00b8fa48539f2db333759ff547006c6818ef895c715e4efeb"} Nov 03 22:58:40.421260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.421089 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:40.422727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.422700 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" event={"ID":"06e66f7f-49d3-4e1d-9557-b8c57159c381","Type":"ContainerStarted","Data":"0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0"} Nov 03 22:58:40.422880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.422750 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" event={"ID":"06e66f7f-49d3-4e1d-9557-b8c57159c381","Type":"ContainerStarted","Data":"4ecc0a13522970682b4f60afa24502807bf66bce5949858ec7e03350175b8743"} Nov 03 22:58:40.422880 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.422804 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:40.540260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.540208 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" podStartSLOduration=1.5401929380000001 podStartE2EDuration="1.540192938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:40.538227231 +0000 UTC m=+2163.750996281" watchObservedRunningTime="2026-11-03 22:58:40.540192938 +0000 UTC m=+2163.752961988" Nov 03 22:58:40.664216 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:40.660867 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" podStartSLOduration=1.660847516 podStartE2EDuration="1.660847516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:40.656998881 +0000 UTC m=+2163.869767954" watchObservedRunningTime="2026-11-03 22:58:40.660847516 +0000 UTC m=+2163.873616582" Nov 03 22:58:46.683150 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:46.683108 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl"] Nov 03 22:58:46.684027 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:46.683970 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" containerID="cri-o://551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70" gracePeriod=30 Nov 03 22:58:49.834847 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:49.834781 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:58:51.441136 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:51.441103 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 22:58:51.441672 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:51.441494 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 22:58:51.606402 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:51.606366 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x"] Nov 03 22:58:51.711023 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:51.710994 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz"] Nov 03 22:58:51.968757 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:51.968696 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.063473 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b996d093-0a87-4e45-afc4-4488b00f78da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b996d093-0a87-4e45-afc4-4488b00f78da-istiod-ca-cert\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.063555 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-socket\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.063615 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-envoy\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.063681 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-credential-socket\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.063763 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-kube-api-access-xkmrc\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-kube-api-access-xkmrc\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.063859 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-certs\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.063996 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b996d093-0a87-4e45-afc4-4488b00f78da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b996d093-0a87-4e45-afc4-4488b00f78da-istio-podinfo\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.064039 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b996d093-0a87-4e45-afc4-4488b00f78da-istiod-ca-cert" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.064070 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-istio-token\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-istio-token\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.064189 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-data\") pod \"b996d093-0a87-4e45-afc4-4488b00f78da\" (UID: \"b996d093-0a87-4e45-afc4-4488b00f78da\") " Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.064914 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b996d093-0a87-4e45-afc4-4488b00f78da-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.065204 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-data" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.065387 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-credential-socket" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.065585 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-socket" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:52.068702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.066629 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-certs" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:52.069419 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.069010 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-envoy" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:52.069864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.069824 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-kube-api-access-xkmrc" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "kube-api-access-xkmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:52.070129 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.070106 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-istio-token" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:52.071427 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.071403 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b996d093-0a87-4e45-afc4-4488b00f78da-istio-podinfo" pod "b996d093-0a87-4e45-afc4-4488b00f78da" (UID: "b996d093-0a87-4e45-afc4-4488b00f78da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:52.165883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165849 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b996d093-0a87-4e45-afc4-4488b00f78da-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.165883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165882 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.165883 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165896 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.166233 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165909 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.166233 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165921 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.166233 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165936 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.166233 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165949 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmrc\" (UniqueName: \"kubernetes.io/projected/b996d093-0a87-4e45-afc4-4488b00f78da-kube-api-access-xkmrc\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.166233 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.165963 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b996d093-0a87-4e45-afc4-4488b00f78da-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:52.521749 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.521714 4898 generic.go:356] "Generic (PLEG): container finished" podID="b996d093-0a87-4e45-afc4-4488b00f78da" containerID="551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70" exitCode=0 Nov 03 22:58:52.522158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.521806 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" Nov 03 22:58:52.522158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.521823 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" event={"ID":"b996d093-0a87-4e45-afc4-4488b00f78da","Type":"ContainerDied","Data":"551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70"} Nov 03 22:58:52.522158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.521895 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl" event={"ID":"b996d093-0a87-4e45-afc4-4488b00f78da","Type":"ContainerDied","Data":"3f17e1f8ae4b7e27518d555bc519c11b69fbc87d05cb62e49089817c6a7a58b9"} Nov 03 22:58:52.522158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.521924 4898 scope.go:122] "RemoveContainer" containerID="551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70" Nov 03 22:58:52.535363 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.535345 4898 scope.go:122] "RemoveContainer" containerID="01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08" Nov 03 22:58:52.544626 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.544604 4898 scope.go:122] "RemoveContainer" containerID="551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70" Nov 03 22:58:52.544987 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:52.544965 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70\": container with ID starting with 551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70 not found: ID does not exist" containerID="551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70" Nov 03 22:58:52.545098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.544994 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70"} err="failed to get container status \"551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70\": rpc error: code = NotFound desc = could not find container \"551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70\": container with ID starting with 551ed554a86175392f05aec8684647da667109e7603d9de7cf57eaee67950a70 not found: ID does not exist" Nov 03 22:58:52.545098 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.545013 4898 scope.go:122] "RemoveContainer" containerID="01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08" Nov 03 22:58:52.545250 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:52.545234 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08\": container with ID starting with 01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08 not found: ID does not exist" containerID="01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08" Nov 03 22:58:52.545309 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.545254 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08"} err="failed to get container status \"01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08\": rpc error: code = NotFound desc = could not find container \"01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08\": container with ID starting with 01e1683eb45a9ea2579cf16f252f26bccb195e07cf37ac0f9283c4b7a0534c08 not found: ID does not exist" Nov 03 22:58:52.689192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.689154 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl"] Nov 03 22:58:52.693206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.693183 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2420f6-openshift-default-75c86ff8ff-ltdfl"] Nov 03 22:58:52.908330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:52.908294 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" path="/var/lib/kubelet/pods/b996d093-0a87-4e45-afc4-4488b00f78da/volumes" Nov 03 22:58:56.606682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:56.606639 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" podUID="495459c3-278c-4bdd-a2cc-6ce3b11023e8" containerName="payload-processing" containerID="cri-o://edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81" gracePeriod=25 Nov 03 22:58:56.712200 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:56.712160 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" podUID="ca719cff-42d6-4515-86fb-c72eca762d1d" containerName="payload-pre-processing" containerID="cri-o://798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87" gracePeriod=25 Nov 03 22:58:56.891863 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:56.891835 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:58:57.001434 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.001406 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:58:57.025076 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.025044 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495459c3-278c-4bdd-a2cc-6ce3b11023e8-kube-api-access-gtzr2\" (UniqueName: \"kubernetes.io/projected/495459c3-278c-4bdd-a2cc-6ce3b11023e8-kube-api-access-gtzr2\") pod \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " Nov 03 22:58:57.025260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.025224 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495459c3-278c-4bdd-a2cc-6ce3b11023e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495459c3-278c-4bdd-a2cc-6ce3b11023e8-plugins-config-volume\") pod \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\" (UID: \"495459c3-278c-4bdd-a2cc-6ce3b11023e8\") " Nov 03 22:58:57.025676 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.025645 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495459c3-278c-4bdd-a2cc-6ce3b11023e8-plugins-config-volume" pod "495459c3-278c-4bdd-a2cc-6ce3b11023e8" (UID: "495459c3-278c-4bdd-a2cc-6ce3b11023e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:57.025899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.025875 4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495459c3-278c-4bdd-a2cc-6ce3b11023e8-plugins-config-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:57.028061 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.028033 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495459c3-278c-4bdd-a2cc-6ce3b11023e8-kube-api-access-gtzr2" pod "495459c3-278c-4bdd-a2cc-6ce3b11023e8" (UID: "495459c3-278c-4bdd-a2cc-6ce3b11023e8"). InnerVolumeSpecName "kube-api-access-gtzr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:57.127034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.126956 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca719cff-42d6-4515-86fb-c72eca762d1d-kube-api-access-qvphk\" (UniqueName: \"kubernetes.io/projected/ca719cff-42d6-4515-86fb-c72eca762d1d-kube-api-access-qvphk\") pod \"ca719cff-42d6-4515-86fb-c72eca762d1d\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " Nov 03 22:58:57.127198 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.127145 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca719cff-42d6-4515-86fb-c72eca762d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca719cff-42d6-4515-86fb-c72eca762d1d-plugins-config-volume\") pod \"ca719cff-42d6-4515-86fb-c72eca762d1d\" (UID: \"ca719cff-42d6-4515-86fb-c72eca762d1d\") " Nov 03 22:58:57.127659 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.127627 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca719cff-42d6-4515-86fb-c72eca762d1d-plugins-config-volume" pod "ca719cff-42d6-4515-86fb-c72eca762d1d" (UID: "ca719cff-42d6-4515-86fb-c72eca762d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:57.127769 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.127667 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzr2\" (UniqueName: \"kubernetes.io/projected/495459c3-278c-4bdd-a2cc-6ce3b11023e8-kube-api-access-gtzr2\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:57.129730 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.129701 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca719cff-42d6-4515-86fb-c72eca762d1d-kube-api-access-qvphk" pod "ca719cff-42d6-4515-86fb-c72eca762d1d" (UID: "ca719cff-42d6-4515-86fb-c72eca762d1d"). InnerVolumeSpecName "kube-api-access-qvphk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:57.229027 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.228996 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvphk\" (UniqueName: \"kubernetes.io/projected/ca719cff-42d6-4515-86fb-c72eca762d1d-kube-api-access-qvphk\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:57.229027 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.229024 4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca719cff-42d6-4515-86fb-c72eca762d1d-plugins-config-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:57.556397 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.556318 4898 generic.go:356] "Generic (PLEG): container finished" podID="495459c3-278c-4bdd-a2cc-6ce3b11023e8" containerID="edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81" exitCode=0 Nov 03 22:58:57.556397 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.556361 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" Nov 03 22:58:57.556601 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.556400 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" event={"ID":"495459c3-278c-4bdd-a2cc-6ce3b11023e8","Type":"ContainerDied","Data":"edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81"} Nov 03 22:58:57.556601 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.556461 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x" event={"ID":"495459c3-278c-4bdd-a2cc-6ce3b11023e8","Type":"ContainerDied","Data":"d348c795dabbf585b73bf59dee468434ec33b61292f4c1255734f9faf7bf3cff"} Nov 03 22:58:57.556601 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.556488 4898 scope.go:122] "RemoveContainer" containerID="edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81" Nov 03 22:58:57.558008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.557981 4898 generic.go:356] "Generic (PLEG): container finished" podID="ca719cff-42d6-4515-86fb-c72eca762d1d" containerID="798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87" exitCode=0 Nov 03 22:58:57.558146 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.558025 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" Nov 03 22:58:57.558146 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.558022 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" event={"ID":"ca719cff-42d6-4515-86fb-c72eca762d1d","Type":"ContainerDied","Data":"798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87"} Nov 03 22:58:57.558273 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.558147 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz" event={"ID":"ca719cff-42d6-4515-86fb-c72eca762d1d","Type":"ContainerDied","Data":"5e0ad725c06eeec21130f9f67259412377bac322ff9a185c690a4b26df29264c"} Nov 03 22:58:57.567226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.567209 4898 scope.go:122] "RemoveContainer" containerID="edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81" Nov 03 22:58:57.567576 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:57.567554 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81\": container with ID starting with edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81 not found: ID does not exist" containerID="edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81" Nov 03 22:58:57.567664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.567582 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81"} err="failed to get container status \"edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81\": rpc error: code = NotFound desc = could not find container \"edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81\": container with ID starting with edb7581df42c722a505669f3b0b640756cfe3ef92060b75d06bf08f7584dbe81 not found: ID does not exist" Nov 03 22:58:57.567664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.567599 4898 scope.go:122] "RemoveContainer" containerID="798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87" Nov 03 22:58:57.577178 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.577162 4898 scope.go:122] "RemoveContainer" containerID="798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87" Nov 03 22:58:57.577455 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:58:57.577434 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87\": container with ID starting with 798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87 not found: ID does not exist" containerID="798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87" Nov 03 22:58:57.577527 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.577461 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87"} err="failed to get container status \"798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87\": rpc error: code = NotFound desc = could not find container \"798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87\": container with ID starting with 798ff8bebd23c585175c4a6dc86cb85fb382c5051a96f2c920dfd5af922b1f87 not found: ID does not exist" Nov 03 22:58:57.731534 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.731498 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz"] Nov 03 22:58:57.780195 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.780164 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2769cb-7668b55d7-b9vqz"] Nov 03 22:58:57.847589 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.847560 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x"] Nov 03 22:58:57.851830 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:57.851803 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2769cb-85cd775f46-hxl4x"] Nov 03 22:58:58.908386 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:58.908350 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495459c3-278c-4bdd-a2cc-6ce3b11023e8" path="/var/lib/kubelet/pods/495459c3-278c-4bdd-a2cc-6ce3b11023e8/volumes" Nov 03 22:58:58.908908 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:58:58.908777 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca719cff-42d6-4515-86fb-c72eca762d1d" path="/var/lib/kubelet/pods/ca719cff-42d6-4515-86fb-c72eca762d1d/volumes" Nov 03 22:59:18.587130 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.587048 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5"] Nov 03 22:59:18.587569 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.587294 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" podUID="b73a40f4-2328-4de8-9ec5-569bd38d8f3b" containerName="maas-api" containerID="cri-o://c8163005d1988ba68f157cca39125c14ebc12c676646943ef87e738936641bb1" gracePeriod=30 Nov 03 22:59:18.715991 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.715953 4898 generic.go:356] "Generic (PLEG): container finished" podID="b73a40f4-2328-4de8-9ec5-569bd38d8f3b" containerID="c8163005d1988ba68f157cca39125c14ebc12c676646943ef87e738936641bb1" exitCode=0 Nov 03 22:59:18.716153 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.716024 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" event={"ID":"b73a40f4-2328-4de8-9ec5-569bd38d8f3b","Type":"ContainerDied","Data":"c8163005d1988ba68f157cca39125c14ebc12c676646943ef87e738936641bb1"} Nov 03 22:59:18.890764 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.890736 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:59:18.991851 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.991781 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-kube-api-access-ptblc\" (UniqueName: \"kubernetes.io/projected/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-kube-api-access-ptblc\") pod \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " Nov 03 22:59:18.992040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.991928 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-openshift-service-ca\") pod \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " Nov 03 22:59:18.992040 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.992030 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-metrics-certs\") pod \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " Nov 03 22:59:18.992187 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.992120 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls\") pod \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\" (UID: \"b73a40f4-2328-4de8-9ec5-569bd38d8f3b\") " Nov 03 22:59:18.995269 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:18.995230 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-openshift-service-ca" pod "b73a40f4-2328-4de8-9ec5-569bd38d8f3b" (UID: "b73a40f4-2328-4de8-9ec5-569bd38d8f3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:19.007280 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.007230 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls" pod "b73a40f4-2328-4de8-9ec5-569bd38d8f3b" (UID: "b73a40f4-2328-4de8-9ec5-569bd38d8f3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:19.007280 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.007254 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-kube-api-access-ptblc" pod "b73a40f4-2328-4de8-9ec5-569bd38d8f3b" (UID: "b73a40f4-2328-4de8-9ec5-569bd38d8f3b"). InnerVolumeSpecName "kube-api-access-ptblc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:19.008586 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.008556 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-metrics-certs" pod "b73a40f4-2328-4de8-9ec5-569bd38d8f3b" (UID: "b73a40f4-2328-4de8-9ec5-569bd38d8f3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:19.094811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.094457 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:19.094811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.094493 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:19.094811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.094511 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptblc\" (UniqueName: \"kubernetes.io/projected/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-kube-api-access-ptblc\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:19.094811 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.094526 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a40f4-2328-4de8-9ec5-569bd38d8f3b-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:19.723408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.723377 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" Nov 03 22:59:19.723408 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.723389 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5" event={"ID":"b73a40f4-2328-4de8-9ec5-569bd38d8f3b","Type":"ContainerDied","Data":"0f9220be72d3459b322a8316710994ea0726c7b2f2628589c66b48eebf9d4ff8"} Nov 03 22:59:19.723976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.723459 4898 scope.go:122] "RemoveContainer" containerID="c8163005d1988ba68f157cca39125c14ebc12c676646943ef87e738936641bb1" Nov 03 22:59:19.836772 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:59:19.836729 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:59:19.892053 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.892019 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5"] Nov 03 22:59:19.895743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:19.895716 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12450a06-68776f84c-k6kr5"] Nov 03 22:59:20.907428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:20.907387 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73a40f4-2328-4de8-9ec5-569bd38d8f3b" path="/var/lib/kubelet/pods/b73a40f4-2328-4de8-9ec5-569bd38d8f3b/volumes" Nov 03 22:59:28.067306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.067270 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf"] Nov 03 22:59:28.068240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068215 4898 cpu_manager.go:414] "containerMap: removing container" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" Nov 03 22:59:28.068240 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068241 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068293 4898 cpu_manager.go:414] "containerMap: removing container" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068302 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068311 4898 cpu_manager.go:414] "containerMap: removing container" podUID="495459c3-278c-4bdd-a2cc-6ce3b11023e8" containerName="payload-processing" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068320 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495459c3-278c-4bdd-a2cc-6ce3b11023e8" containerName="payload-processing" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068336 4898 cpu_manager.go:414] "containerMap: removing container" podUID="ca719cff-42d6-4515-86fb-c72eca762d1d" containerName="payload-pre-processing" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068343 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca719cff-42d6-4515-86fb-c72eca762d1d" containerName="payload-pre-processing" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068381 4898 cpu_manager.go:414] "containerMap: removing container" podUID="b73a40f4-2328-4de8-9ec5-569bd38d8f3b" containerName="maas-api" Nov 03 22:59:28.068438 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068389 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73a40f4-2328-4de8-9ec5-569bd38d8f3b" containerName="maas-api" Nov 03 22:59:28.068870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068555 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73a40f4-2328-4de8-9ec5-569bd38d8f3b" containerName="maas-api" Nov 03 22:59:28.068870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068570 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" Nov 03 22:59:28.068870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068585 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="495459c3-278c-4bdd-a2cc-6ce3b11023e8" containerName="payload-processing" Nov 03 22:59:28.068870 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.068598 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca719cff-42d6-4515-86fb-c72eca762d1d" containerName="payload-pre-processing" Nov 03 22:59:28.074866 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.074845 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103116 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-envoy\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103155 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103170 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-credential-socket\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103206 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-socket\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103229 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-token\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103301 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-data\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103325 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7k2\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-kube-api-access-jr7k2\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103408 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-certs\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103440 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-podinfo\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.103590 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.103471 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istiod-ca-cert\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.128428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.128398 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2cea18-openshift-default-dockercfg-vqjbd\"" Nov 03 22:59:28.186826 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.183106 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf"] Nov 03 22:59:28.213001 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.211131 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-certs\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.213330 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.213309 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-podinfo\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.213942 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.213916 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istiod-ca-cert\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.214888 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.214867 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-envoy\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.215484 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.215466 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-credential-socket\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.215621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.215608 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-socket\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.215734 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.215720 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-token\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.215976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.215955 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-data\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.216116 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.216098 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7k2\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-kube-api-access-jr7k2\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.216575 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.214734 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istiod-ca-cert\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.216985 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.216964 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-credential-socket\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.217360 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.217340 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-socket\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.239850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.220191 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-data\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.239850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.213216 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-certs\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.239850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.225577 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-podinfo\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.239850 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.227418 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-envoy\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.280066 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.280026 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7k2\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-kube-api-access-jr7k2\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.305574 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.305534 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-token\") pod \"e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.390143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.390059 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:28.586517 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.586478 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l"] Nov 03 22:59:28.587227 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.587210 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b996d093-0a87-4e45-afc4-4488b00f78da" containerName="istio-proxy" Nov 03 22:59:28.591840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.591781 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.602205 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.602179 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc"] Nov 03 22:59:28.606059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.606039 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.625265 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625240 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.625387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625271 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.625387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625291 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.625387 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625318 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.625514 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625468 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.625514 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625499 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.625604 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625547 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.625604 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625598 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.625721 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625679 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24779e64-aabc-4fa5-8701-79d557ea9c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.625776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625717 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.625776 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625746 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8vl\" (UniqueName: \"kubernetes.io/projected/a8441d29-d512-4219-af7f-3a27ccbfd3fa-kube-api-access-sf8vl\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.625894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625850 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscft\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-kube-api-access-dscft\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.625894 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625876 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.626166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625912 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.626166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625969 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.626166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.625989 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.626166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.626075 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.626166 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.626135 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.653682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.653163 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l"] Nov 03 22:59:28.701885 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.701853 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5vrs\"" Nov 03 22:59:28.722907 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.721778 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc"] Nov 03 22:59:28.738075 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.738036 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscft\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-kube-api-access-dscft\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.738341 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.738321 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.738493 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.738479 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.738602 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.738589 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.738710 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.738696 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.738856 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.738841 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.739000 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.738986 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.739105 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739093 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.739199 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739186 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.739299 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739285 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.739414 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739401 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.739535 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739519 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.739628 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739615 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.739769 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739756 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.739934 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.739918 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.740079 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.740062 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24779e64-aabc-4fa5-8701-79d557ea9c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.740245 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.740193 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.740380 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.740352 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8vl\" (UniqueName: \"kubernetes.io/projected/a8441d29-d512-4219-af7f-3a27ccbfd3fa-kube-api-access-sf8vl\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.741566 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.741533 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24779e64-aabc-4fa5-8701-79d557ea9c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.741692 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.741667 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.743214 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.742624 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.743214 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.742896 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.744206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.744162 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.744587 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.744561 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.746119 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.746088 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.746659 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.746634 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.747414 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.747387 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.749258 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.748933 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.750622 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.750602 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.751591 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.751544 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.755258 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.755232 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.756374 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.756032 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.802050 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.802013 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscft\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-kube-api-access-dscft\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.802243 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.802096 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8vl\" (UniqueName: \"kubernetes.io/projected/a8441d29-d512-4219-af7f-3a27ccbfd3fa-kube-api-access-sf8vl\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.806512 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.806485 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jlhrc\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:28.809260 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.809242 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8441d29-d512-4219-af7f-3a27ccbfd3fa-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j6q6l\" (UID: \"a8441d29-d512-4219-af7f-3a27ccbfd3fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.837203 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:59:28.837169 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909a0d33_8e64_42a5_9ed0_67cb0a96a413.slice/crio-61551284e4c17b6066fae6973e4ca63c1de6a77474fc10314ffa9e3d394560f5 WatchSource:0}: Error finding container 61551284e4c17b6066fae6973e4ca63c1de6a77474fc10314ffa9e3d394560f5: Status 404 returned error can't find the container with id 61551284e4c17b6066fae6973e4ca63c1de6a77474fc10314ffa9e3d394560f5 Nov 03 22:59:28.839525 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.839493 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf"] Nov 03 22:59:28.840131 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.840099 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:28.840220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.840160 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:28.840220 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.840189 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:28.907948 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.907884 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:28.919735 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:28.919708 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:29.319059 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.319028 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l"] Nov 03 22:59:29.320242 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:59:29.320212 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8441d29_d512_4219_af7f_3a27ccbfd3fa.slice/crio-d44411c3d1d819f45548a8e5462d085d2f2984fd5c17c38b48fb453d88c13c2c WatchSource:0}: Error finding container d44411c3d1d819f45548a8e5462d085d2f2984fd5c17c38b48fb453d88c13c2c: Status 404 returned error can't find the container with id d44411c3d1d819f45548a8e5462d085d2f2984fd5c17c38b48fb453d88c13c2c Nov 03 22:59:29.323454 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.323411 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:29.323552 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.323532 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:29.323594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.323583 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:29.369030 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.368999 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc"] Nov 03 22:59:29.372375 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:59:29.372349 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24779e64_aabc_4fa5_8701_79d557ea9c32.slice/crio-f7594f989b2162b0ef1bbefcda96d78223b33b3c3d85907bbf3f70f5c26a4d89 WatchSource:0}: Error finding container f7594f989b2162b0ef1bbefcda96d78223b33b3c3d85907bbf3f70f5c26a4d89: Status 404 returned error can't find the container with id f7594f989b2162b0ef1bbefcda96d78223b33b3c3d85907bbf3f70f5c26a4d89 Nov 03 22:59:29.375486 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.375448 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:29.375606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.375534 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:29.375606 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.375577 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:29.800649 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.800553 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" event={"ID":"909a0d33-8e64-42a5-9ed0-67cb0a96a413","Type":"ContainerStarted","Data":"1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f"} Nov 03 22:59:29.800649 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.800619 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" event={"ID":"909a0d33-8e64-42a5-9ed0-67cb0a96a413","Type":"ContainerStarted","Data":"61551284e4c17b6066fae6973e4ca63c1de6a77474fc10314ffa9e3d394560f5"} Nov 03 22:59:29.802826 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.802767 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" event={"ID":"24779e64-aabc-4fa5-8701-79d557ea9c32","Type":"ContainerStarted","Data":"3e43701edda68272023e0a9978274003e8efe52f159f3ac7b8bc2d9f3eded88e"} Nov 03 22:59:29.802972 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.802850 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" event={"ID":"24779e64-aabc-4fa5-8701-79d557ea9c32","Type":"ContainerStarted","Data":"f7594f989b2162b0ef1bbefcda96d78223b33b3c3d85907bbf3f70f5c26a4d89"} Nov 03 22:59:29.804804 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.804763 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" event={"ID":"a8441d29-d512-4219-af7f-3a27ccbfd3fa","Type":"ContainerStarted","Data":"fb464e8cb53015c4e7cb22001d643305e27c4dade934c2237d8854562127ed5b"} Nov 03 22:59:29.804928 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.804824 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" event={"ID":"a8441d29-d512-4219-af7f-3a27ccbfd3fa","Type":"ContainerStarted","Data":"d44411c3d1d819f45548a8e5462d085d2f2984fd5c17c38b48fb453d88c13c2c"} Nov 03 22:59:29.908336 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.908294 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:29.920670 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.920643 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:29.921842 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:29.921776 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podStartSLOduration=1.921762904 podStartE2EDuration="1.921762904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:29.921433024 +0000 UTC m=+2213.134202078" watchObservedRunningTime="2026-11-03 22:59:29.921762904 +0000 UTC m=+2213.134531954" Nov 03 22:59:30.054469 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.054350 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podStartSLOduration=2.054328763 podStartE2EDuration="2.054328763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:30.053951685 +0000 UTC m=+2213.266720738" watchObservedRunningTime="2026-11-03 22:59:30.054328763 +0000 UTC m=+2213.267097816" Nov 03 22:59:30.179662 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.179595 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podStartSLOduration=2.179576596 podStartE2EDuration="2.179576596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:30.176113108 +0000 UTC m=+2213.388882180" watchObservedRunningTime="2026-11-03 22:59:30.179576596 +0000 UTC m=+2213.392345648" Nov 03 22:59:30.391006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.390964 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:30.397571 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.397540 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:30.824533 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.824501 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:30.826192 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.826172 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 22:59:30.911777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.911424 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:30.911777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.911491 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:30.921297 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.921263 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jlhrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:59:30.921489 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:30.921309 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:59:31.908661 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:31.908619 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:31.909107 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:31.908673 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:31.920291 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:31.920264 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jlhrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:59:31.920420 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:31.920313 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:59:32.909271 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:32.909236 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:32.909819 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:32.909775 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:32.920540 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:32.920518 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jlhrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:59:32.920689 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:32.920671 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:59:33.908497 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:33.908458 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:33.908683 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:33.908511 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:33.921358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:33.921237 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jlhrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:59:33.921358 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:33.921299 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:59:34.908933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:34.908884 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:34.909143 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:34.908951 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:34.920487 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:34.920456 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jlhrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:59:34.920599 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:34.920504 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:59:35.908831 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:35.908775 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:35.909295 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:35.908854 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:35.921146 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:35.921118 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jlhrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:59:35.921276 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:35.921169 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:59:36.908969 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:36.908936 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:36.909441 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:36.908991 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:36.920747 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:36.920712 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jlhrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:59:36.920879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:36.920827 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:59:37.917818 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:37.910178 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:59:37.917818 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:37.910272 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:59:38.795939 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:38.795904 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:38.796736 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:38.796710 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:38.797642 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:38.797617 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 22:59:39.281502 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:39.281460 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:39.283030 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:39.283003 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:39.283742 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:39.283717 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" Nov 03 22:59:43.728970 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.728933 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j"] Nov 03 22:59:43.734523 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.734500 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.779381 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.779344 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh"] Nov 03 22:59:43.783483 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.783461 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.806006 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.805982 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806131 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806019 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806131 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806043 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806203 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806306 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806243 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806413 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806331 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbfw\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-kube-api-access-pfbfw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806413 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806372 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78915b29-4a76-4c77-8735-e67fa51c3fc4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806482 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.806520 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.806511 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.832815 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.832766 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j"] Nov 03 22:59:43.898149 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.898116 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh"] Nov 03 22:59:43.908262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908230 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.908519 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908492 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbfw\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-kube-api-access-pfbfw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.908621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908538 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78915b29-4a76-4c77-8735-e67fa51c3fc4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.908621 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908614 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.908739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908639 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.908739 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908710 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.908879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908775 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75gln\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-kube-api-access-75gln\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.908879 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908836 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.908994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908883 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.908994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908899 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.908994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908909 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.908994 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.908640 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.909263 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909239 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.909391 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909366 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.909518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909406 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.909518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909435 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.909518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909462 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.909518 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909496 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.909743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909525 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.909743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909571 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85456a7b-db98-4291-b3ed-e01fa8d7052f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:43.909743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909593 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78915b29-4a76-4c77-8735-e67fa51c3fc4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.909743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909607 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.910008 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.909986 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.911718 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.911694 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.912613 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.912586 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.970953 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.970913 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:43.971342 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:43.971314 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbfw\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-kube-api-access-pfbfw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-std4j\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:44.011352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011268 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011352 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011341 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75gln\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-kube-api-access-75gln\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011389 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011431 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011478 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011507 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011551 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011539 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011566 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011611 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85456a7b-db98-4291-b3ed-e01fa8d7052f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.011837 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011775 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.012020 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.011997 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.012435 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.012408 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85456a7b-db98-4291-b3ed-e01fa8d7052f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.012604 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.012557 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.012702 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.012608 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.014325 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.014297 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.014644 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.014624 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.048594 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.048572 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:44.073595 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.073570 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.073893 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.073803 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75gln\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-kube-api-access-75gln\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nr7zh\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.096631 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.096601 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:44.485652 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.485619 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j"] Nov 03 22:59:44.486461 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:59:44.486430 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78915b29_4a76_4c77_8735_e67fa51c3fc4.slice/crio-d2b184991139b5e9abaef678fdbdef3e4b9e6e8de3d1e241e682fda668039f9b WatchSource:0}: Error finding container d2b184991139b5e9abaef678fdbdef3e4b9e6e8de3d1e241e682fda668039f9b: Status 404 returned error can't find the container with id d2b184991139b5e9abaef678fdbdef3e4b9e6e8de3d1e241e682fda668039f9b Nov 03 22:59:44.489226 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.489191 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:44.489304 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.489253 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:44.489304 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.489281 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:44.539329 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.539302 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh"] Nov 03 22:59:44.540869 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:59:44.540845 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85456a7b_db98_4291_b3ed_e01fa8d7052f.slice/crio-73d2318ebd7311f98da1067c5931cb25b00c94d591a2957fb18f27f12875596c WatchSource:0}: Error finding container 73d2318ebd7311f98da1067c5931cb25b00c94d591a2957fb18f27f12875596c: Status 404 returned error can't find the container with id 73d2318ebd7311f98da1067c5931cb25b00c94d591a2957fb18f27f12875596c Nov 03 22:59:44.544080 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.544051 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:44.544144 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.544112 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:44.544181 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:44.544143 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:45.113060 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:45.113016 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" event={"ID":"85456a7b-db98-4291-b3ed-e01fa8d7052f","Type":"ContainerStarted","Data":"3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119"} Nov 03 22:59:45.113508 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:45.113155 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" event={"ID":"85456a7b-db98-4291-b3ed-e01fa8d7052f","Type":"ContainerStarted","Data":"73d2318ebd7311f98da1067c5931cb25b00c94d591a2957fb18f27f12875596c"} Nov 03 22:59:45.114628 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:45.114595 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" event={"ID":"78915b29-4a76-4c77-8735-e67fa51c3fc4","Type":"ContainerStarted","Data":"359e7cdf125bf78cb930d355536146df3daf61b891a17e221387b0099259208e"} Nov 03 22:59:45.114758 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:45.114645 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" event={"ID":"78915b29-4a76-4c77-8735-e67fa51c3fc4","Type":"ContainerStarted","Data":"d2b184991139b5e9abaef678fdbdef3e4b9e6e8de3d1e241e682fda668039f9b"} Nov 03 22:59:45.272812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:45.272574 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podStartSLOduration=2.272539514 podStartE2EDuration="2.272539514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:45.259091638 +0000 UTC m=+2228.471860693" watchObservedRunningTime="2026-11-03 22:59:45.272539514 +0000 UTC m=+2228.485308567" Nov 03 22:59:45.397871 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:45.397053 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podStartSLOduration=2.397029246 podStartE2EDuration="2.397029246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:45.388268281 +0000 UTC m=+2228.601037333" watchObservedRunningTime="2026-11-03 22:59:45.397029246 +0000 UTC m=+2228.609798308" Nov 03 22:59:46.054252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.050086 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:46.064810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.055427 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:59:46.064810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.055484 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:59:46.098824 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.097094 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:46.099015 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.098944 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:59:46.099015 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.098995 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 22:59:46.806564 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.806520 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f"] Nov 03 22:59:46.807428 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:46.807388 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" podUID="a76a7d14-6ee0-4ce0-a090-4249044af855" containerName="maas-api" containerID="cri-o://77fdfde5d2b267754a7ea8f5c93d70bd7fb6662b280c447d8f6e2a965ac6a941" gracePeriod=30 Nov 03 22:59:47.058867 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.054317 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:59:47.058867 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.054396 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:59:47.106810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.106673 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:59:47.106810 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.106735 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 22:59:47.188095 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.188058 4898 generic.go:356] "Generic (PLEG): container finished" podID="a76a7d14-6ee0-4ce0-a090-4249044af855" containerID="77fdfde5d2b267754a7ea8f5c93d70bd7fb6662b280c447d8f6e2a965ac6a941" exitCode=0 Nov 03 22:59:47.207812 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.189298 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" event={"ID":"a76a7d14-6ee0-4ce0-a090-4249044af855","Type":"ContainerDied","Data":"77fdfde5d2b267754a7ea8f5c93d70bd7fb6662b280c447d8f6e2a965ac6a941"} Nov 03 22:59:47.798946 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.775217 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:59:47.880266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.879104 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76a7d14-6ee0-4ce0-a090-4249044af855-kube-api-access-rldn5\" (UniqueName: \"kubernetes.io/projected/a76a7d14-6ee0-4ce0-a090-4249044af855-kube-api-access-rldn5\") pod \"a76a7d14-6ee0-4ce0-a090-4249044af855\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " Nov 03 22:59:47.880266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.879168 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls\") pod \"a76a7d14-6ee0-4ce0-a090-4249044af855\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " Nov 03 22:59:47.880266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.879244 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-metrics-certs\") pod \"a76a7d14-6ee0-4ce0-a090-4249044af855\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " Nov 03 22:59:47.880266 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.879331 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76a7d14-6ee0-4ce0-a090-4249044af855-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76a7d14-6ee0-4ce0-a090-4249044af855-openshift-service-ca\") pod \"a76a7d14-6ee0-4ce0-a090-4249044af855\" (UID: \"a76a7d14-6ee0-4ce0-a090-4249044af855\") " Nov 03 22:59:47.881158 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.881129 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76a7d14-6ee0-4ce0-a090-4249044af855-openshift-service-ca" pod "a76a7d14-6ee0-4ce0-a090-4249044af855" (UID: "a76a7d14-6ee0-4ce0-a090-4249044af855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:47.930138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.929926 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls" pod "a76a7d14-6ee0-4ce0-a090-4249044af855" (UID: "a76a7d14-6ee0-4ce0-a090-4249044af855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:47.930138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.930056 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76a7d14-6ee0-4ce0-a090-4249044af855-kube-api-access-rldn5" pod "a76a7d14-6ee0-4ce0-a090-4249044af855" (UID: "a76a7d14-6ee0-4ce0-a090-4249044af855"). InnerVolumeSpecName "kube-api-access-rldn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:47.930138 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.930109 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-metrics-certs" pod "a76a7d14-6ee0-4ce0-a090-4249044af855" (UID: "a76a7d14-6ee0-4ce0-a090-4249044af855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:47.983111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.983003 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rldn5\" (UniqueName: \"kubernetes.io/projected/a76a7d14-6ee0-4ce0-a090-4249044af855-kube-api-access-rldn5\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:47.983111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.983041 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:47.983111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.983055 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76a7d14-6ee0-4ce0-a090-4249044af855-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:47.983111 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:47.983069 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76a7d14-6ee0-4ce0-a090-4249044af855-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:48.086899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.063353 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:59:48.086899 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.063415 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:59:48.107933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.107398 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:59:48.107933 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.107458 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 22:59:48.217727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.217693 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" Nov 03 22:59:48.217727 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.217716 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f" event={"ID":"a76a7d14-6ee0-4ce0-a090-4249044af855","Type":"ContainerDied","Data":"a8ef296531062c4733952b619c6c10751be53a0ce057be8e26fd124c4a7b72fe"} Nov 03 22:59:48.217976 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.217813 4898 scope.go:122] "RemoveContainer" containerID="77fdfde5d2b267754a7ea8f5c93d70bd7fb6662b280c447d8f6e2a965ac6a941" Nov 03 22:59:48.446640 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.446539 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f"] Nov 03 22:59:48.477820 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.472196 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2627c5-77976c88c4-49d5f"] Nov 03 22:59:48.909200 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:48.909168 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76a7d14-6ee0-4ce0-a090-4249044af855" path="/var/lib/kubelet/pods/a76a7d14-6ee0-4ce0-a090-4249044af855/volumes" Nov 03 22:59:49.054956 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:49.054919 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:59:49.055214 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:49.055182 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:59:49.097556 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:49.097518 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:59:49.097864 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:49.097831 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 22:59:49.839657 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:59:49.839620 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 22:59:50.052197 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:50.052156 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:59:50.052781 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:50.052749 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:59:50.098210 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:50.098123 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:59:50.098446 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:50.098421 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 22:59:51.049010 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:51.048968 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:59:51.049201 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:51.049039 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:59:51.098271 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:51.098227 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:59:51.098647 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:51.098299 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 22:59:52.049403 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:52.049361 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:59:52.049615 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:52.049428 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:59:52.097583 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:52.097540 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:59:52.097750 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:52.097606 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 22:59:53.117565 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.117526 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm"] Nov 03 22:59:53.118394 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.118358 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" containerID="cri-o://aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec" gracePeriod=30 Nov 03 22:59:53.625034 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.624997 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:53.626239 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.626217 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:53.627113 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.627088 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 22:59:53.780366 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.780332 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:53.781347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.781202 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:53.782596 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:53.782573 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 22:59:54.313124 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.313095 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k"] Nov 03 22:59:54.313696 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.313682 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a76a7d14-6ee0-4ce0-a090-4249044af855" containerName="maas-api" Nov 03 22:59:54.313743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.313697 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76a7d14-6ee0-4ce0-a090-4249044af855" containerName="maas-api" Nov 03 22:59:54.313847 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.313836 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76a7d14-6ee0-4ce0-a090-4249044af855" containerName="maas-api" Nov 03 22:59:54.319245 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.319219 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.382035 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.382002 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-31ad7b-serving-cert\"" Nov 03 22:59:54.383959 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.383925 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k"] Nov 03 22:59:54.389516 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.389484 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.389657 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.389617 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsrsg\" (UniqueName: \"kubernetes.io/projected/f5006375-cd5b-4d8e-832b-da705da48540-kube-api-access-nsrsg\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.389857 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.389840 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5006375-cd5b-4d8e-832b-da705da48540-openshift-service-ca\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.389958 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.389939 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-metrics-certs\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.491353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.491304 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5006375-cd5b-4d8e-832b-da705da48540-openshift-service-ca\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.491558 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.491422 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-metrics-certs\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.491558 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.491490 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.491682 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.491568 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsrsg\" (UniqueName: \"kubernetes.io/projected/f5006375-cd5b-4d8e-832b-da705da48540-kube-api-access-nsrsg\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.491682 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:59:54.491575 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-serving-cert: secret "maas-api-e2e-api-a-31ad7b-serving-cert" not found Nov 03 22:59:54.491848 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:59:54.491827 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls podName:f5006375-cd5b-4d8e-832b-da705da48540 nodeName:}" failed. No retries permitted until 2026-11-03 22:59:54.991779702 +0000 UTC m=+2238.204548748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls") pod "maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" (UID: "f5006375-cd5b-4d8e-832b-da705da48540") : secret "maas-api-e2e-api-a-31ad7b-serving-cert" not found Nov 03 22:59:54.492252 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.492230 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5006375-cd5b-4d8e-832b-da705da48540-openshift-service-ca\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.494096 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.494079 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-metrics-certs\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.553840 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.553772 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsrsg\" (UniqueName: \"kubernetes.io/projected/f5006375-cd5b-4d8e-832b-da705da48540-kube-api-access-nsrsg\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:54.999543 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:54.999506 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:55.002067 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:55.002045 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls\") pod \"maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:55.116222 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:55.116183 4898 patch_prober.go:29] interesting pod/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:59:55.116379 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:55.116250 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:59:55.234206 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:55.234171 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:55.621225 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:55.621196 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k"] Nov 03 22:59:55.622134 ip-10-0-139-173 kubenswrapper[4898]: W1103 22:59:55.622105 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5006375_cd5b_4d8e_832b_da705da48540.slice/crio-d2d03257268992ca3102f95e0844f91b95e7d8d4b4204fb235a44f6994203d29 WatchSource:0}: Error finding container d2d03257268992ca3102f95e0844f91b95e7d8d4b4204fb235a44f6994203d29: Status 404 returned error can't find the container with id d2d03257268992ca3102f95e0844f91b95e7d8d4b4204fb235a44f6994203d29 Nov 03 22:59:56.416436 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:56.416403 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" event={"ID":"f5006375-cd5b-4d8e-832b-da705da48540","Type":"ContainerStarted","Data":"d2d03257268992ca3102f95e0844f91b95e7d8d4b4204fb235a44f6994203d29"} Nov 03 22:59:57.424989 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:57.424951 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" event={"ID":"f5006375-cd5b-4d8e-832b-da705da48540","Type":"ContainerStarted","Data":"8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40"} Nov 03 22:59:57.425347 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:57.425014 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 22:59:57.544531 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:57.544242 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" podStartSLOduration=2.533105492 podStartE2EDuration="3.544222394s" totalImagesPullingTime="1.011116902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:57.542495208 +0000 UTC m=+2240.755264261" watchObservedRunningTime="2026-11-03 22:59:57.544222394 +0000 UTC m=+2240.756991446" Nov 03 22:59:58.409145 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.409078 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:59:58.432875 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.432840 4898 generic.go:356] "Generic (PLEG): container finished" podID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerID="aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec" exitCode=0 Nov 03 22:59:58.433262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.432889 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" Nov 03 22:59:58.433262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.432905 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" event={"ID":"08b55968-7e4c-4508-b0b4-a64ad6eeb74d","Type":"ContainerDied","Data":"aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec"} Nov 03 22:59:58.433262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.432958 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm" event={"ID":"08b55968-7e4c-4508-b0b4-a64ad6eeb74d","Type":"ContainerDied","Data":"170e41aee369f156d75a8b8333cb2eea737ff9c61ad65f04ec30d6a35b605b8c"} Nov 03 22:59:58.433262 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.432981 4898 scope.go:122] "RemoveContainer" containerID="aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec" Nov 03 22:59:58.444418 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444387 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-socket\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.444548 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444452 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-credential-socket\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.444548 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444504 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-certs\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.444664 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444632 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-socket" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:58.444720 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444675 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-token\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-token\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.444773 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444742 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-podinfo\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.444773 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444739 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-credential-socket" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:58.444920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444821 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istiod-ca-cert\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.444920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444841 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-certs" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:58.444920 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444867 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-envoy\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.445077 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444925 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-data\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.445077 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.444955 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-kube-api-access-42t5x\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-kube-api-access-42t5x\") pod \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\" (UID: \"08b55968-7e4c-4508-b0b4-a64ad6eeb74d\") " Nov 03 22:59:58.445257 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445227 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-data" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:58.445353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445288 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istiod-ca-cert" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:58.445636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445582 4898 scope.go:122] "RemoveContainer" containerID="922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e" Nov 03 22:59:58.445636 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445585 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.445777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445668 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.445777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445689 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.445777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445704 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.445777 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.445719 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.447624 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.447584 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-envoy" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:58.447832 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.447691 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-token" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:58.448250 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.448228 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-podinfo" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:58.450224 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.450194 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-kube-api-access-42t5x" pod "08b55968-7e4c-4508-b0b4-a64ad6eeb74d" (UID: "08b55968-7e4c-4508-b0b4-a64ad6eeb74d"). InnerVolumeSpecName "kube-api-access-42t5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:58.477020 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.476991 4898 scope.go:122] "RemoveContainer" containerID="aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec" Nov 03 22:59:58.477369 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:59:58.477350 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec\": container with ID starting with aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec not found: ID does not exist" containerID="aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec" Nov 03 22:59:58.477448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.477381 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec"} err="failed to get container status \"aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec\": rpc error: code = NotFound desc = could not find container \"aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec\": container with ID starting with aed9aa04f596691ff8cfc9323d44b2a836300850b25a7643659c71e147ad10ec not found: ID does not exist" Nov 03 22:59:58.477448 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.477401 4898 scope.go:122] "RemoveContainer" containerID="922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e" Nov 03 22:59:58.477676 ip-10-0-139-173 kubenswrapper[4898]: E1103 22:59:58.477654 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e\": container with ID starting with 922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e not found: ID does not exist" containerID="922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e" Nov 03 22:59:58.477743 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.477682 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e"} err="failed to get container status \"922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e\": rpc error: code = NotFound desc = could not find container \"922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e\": container with ID starting with 922de2e93ecd8d47016399aef55367c22022638fd7ac4d9ce1572e136b27241e not found: ID does not exist" Nov 03 22:59:58.547308 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.546983 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.547308 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.547019 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.547308 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.547038 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.547308 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.547056 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42t5x\" (UniqueName: \"kubernetes.io/projected/08b55968-7e4c-4508-b0b4-a64ad6eeb74d-kube-api-access-42t5x\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.864293 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.864256 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm"] Nov 03 22:59:58.873616 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.873578 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2627c5-openshift-default-6c84fbdb55-5g5zm"] Nov 03 22:59:58.907353 ip-10-0-139-173 kubenswrapper[4898]: I1103 22:59:58.907312 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" path="/var/lib/kubelet/pods/08b55968-7e4c-4508-b0b4-a64ad6eeb74d/volumes" Nov 03 23:00:03.439459 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:03.439430 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 23:00:04.457258 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.457222 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8"] Nov 03 23:00:04.457698 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.457441 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" podUID="42ed0f01-5467-4f3c-b635-26f79ef2bd59" containerName="maas-api" containerID="cri-o://f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb" gracePeriod=30 Nov 03 23:00:04.803057 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.803028 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 23:00:04.941309 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.940752 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ed0f01-5467-4f3c-b635-26f79ef2bd59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ed0f01-5467-4f3c-b635-26f79ef2bd59-openshift-service-ca\") pod \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " Nov 03 23:00:04.941497 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.941266 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ed0f01-5467-4f3c-b635-26f79ef2bd59-openshift-service-ca" pod "42ed0f01-5467-4f3c-b635-26f79ef2bd59" (UID: "42ed0f01-5467-4f3c-b635-26f79ef2bd59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:04.941497 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.941409 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls\") pod \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " Nov 03 23:00:04.941613 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.941567 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ed0f01-5467-4f3c-b635-26f79ef2bd59-kube-api-access-2rq5k\" (UniqueName: \"kubernetes.io/projected/42ed0f01-5467-4f3c-b635-26f79ef2bd59-kube-api-access-2rq5k\") pod \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " Nov 03 23:00:04.941679 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.941629 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-metrics-certs\") pod \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\" (UID: \"42ed0f01-5467-4f3c-b635-26f79ef2bd59\") " Nov 03 23:00:04.942269 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.942247 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ed0f01-5467-4f3c-b635-26f79ef2bd59-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:04.944817 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.944757 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-metrics-certs" pod "42ed0f01-5467-4f3c-b635-26f79ef2bd59" (UID: "42ed0f01-5467-4f3c-b635-26f79ef2bd59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:04.964464 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.964425 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls" pod "42ed0f01-5467-4f3c-b635-26f79ef2bd59" (UID: "42ed0f01-5467-4f3c-b635-26f79ef2bd59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:04.966260 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:04.965998 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ed0f01-5467-4f3c-b635-26f79ef2bd59-kube-api-access-2rq5k" pod "42ed0f01-5467-4f3c-b635-26f79ef2bd59" (UID: "42ed0f01-5467-4f3c-b635-26f79ef2bd59"). InnerVolumeSpecName "kube-api-access-2rq5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:05.054591 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.043625 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:05.054591 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.046891 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rq5k\" (UniqueName: \"kubernetes.io/projected/42ed0f01-5467-4f3c-b635-26f79ef2bd59-kube-api-access-2rq5k\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:05.054591 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.046931 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ed0f01-5467-4f3c-b635-26f79ef2bd59-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:05.520928 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.520889 4898 generic.go:356] "Generic (PLEG): container finished" podID="42ed0f01-5467-4f3c-b635-26f79ef2bd59" containerID="f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb" exitCode=0 Nov 03 23:00:05.521564 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.521528 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" event={"ID":"42ed0f01-5467-4f3c-b635-26f79ef2bd59","Type":"ContainerDied","Data":"f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb"} Nov 03 23:00:05.521709 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.521600 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" event={"ID":"42ed0f01-5467-4f3c-b635-26f79ef2bd59","Type":"ContainerDied","Data":"417a3d805d33814da6244ba2174c7fff83b4a6c0d257fa845358717f137ab831"} Nov 03 23:00:05.521709 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.521633 4898 scope.go:122] "RemoveContainer" containerID="f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb" Nov 03 23:00:05.521906 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.521778 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8" Nov 03 23:00:05.556065 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.555950 4898 scope.go:122] "RemoveContainer" containerID="f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb" Nov 03 23:00:05.556586 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:05.556507 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb\": container with ID starting with f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb not found: ID does not exist" containerID="f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb" Nov 03 23:00:05.556586 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.556549 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb"} err="failed to get container status \"f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb\": rpc error: code = NotFound desc = could not find container \"f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb\": container with ID starting with f502cc919be915e23040f4117aa1e47b02df28caab2d12f01d6d256704bcd7bb not found: ID does not exist" Nov 03 23:00:05.701551 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.701517 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8"] Nov 03 23:00:05.705275 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.705251 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc"] Nov 03 23:00:05.705461 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.705441 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" podUID="6c89e2d9-6024-479f-8022-6bdda00c9c02" containerName="maas-api" containerID="cri-o://3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a" gracePeriod=30 Nov 03 23:00:05.710016 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:05.709990 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2769cb-6c44bf64d9-whsm8"] Nov 03 23:00:06.083487 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.083456 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 23:00:06.173814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.173358 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c89e2d9-6024-479f-8022-6bdda00c9c02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c89e2d9-6024-479f-8022-6bdda00c9c02-openshift-service-ca\") pod \"6c89e2d9-6024-479f-8022-6bdda00c9c02\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " Nov 03 23:00:06.173814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.173419 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls\") pod \"6c89e2d9-6024-479f-8022-6bdda00c9c02\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " Nov 03 23:00:06.173814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.173455 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-metrics-certs\") pod \"6c89e2d9-6024-479f-8022-6bdda00c9c02\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " Nov 03 23:00:06.173814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.173555 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c89e2d9-6024-479f-8022-6bdda00c9c02-kube-api-access-hd28m\" (UniqueName: \"kubernetes.io/projected/6c89e2d9-6024-479f-8022-6bdda00c9c02-kube-api-access-hd28m\") pod \"6c89e2d9-6024-479f-8022-6bdda00c9c02\" (UID: \"6c89e2d9-6024-479f-8022-6bdda00c9c02\") " Nov 03 23:00:06.176897 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.176866 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c89e2d9-6024-479f-8022-6bdda00c9c02-openshift-service-ca" pod "6c89e2d9-6024-479f-8022-6bdda00c9c02" (UID: "6c89e2d9-6024-479f-8022-6bdda00c9c02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:06.211696 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.209865 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c89e2d9-6024-479f-8022-6bdda00c9c02-kube-api-access-hd28m" pod "6c89e2d9-6024-479f-8022-6bdda00c9c02" (UID: "6c89e2d9-6024-479f-8022-6bdda00c9c02"). InnerVolumeSpecName "kube-api-access-hd28m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:06.211696 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.210004 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls" pod "6c89e2d9-6024-479f-8022-6bdda00c9c02" (UID: "6c89e2d9-6024-479f-8022-6bdda00c9c02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:06.216865 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.216834 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-metrics-certs" pod "6c89e2d9-6024-479f-8022-6bdda00c9c02" (UID: "6c89e2d9-6024-479f-8022-6bdda00c9c02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:06.276809 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.276745 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c89e2d9-6024-479f-8022-6bdda00c9c02-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:06.276809 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.276802 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:06.276809 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.276817 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c89e2d9-6024-479f-8022-6bdda00c9c02-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:06.277099 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.276832 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd28m\" (UniqueName: \"kubernetes.io/projected/6c89e2d9-6024-479f-8022-6bdda00c9c02-kube-api-access-hd28m\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:06.529426 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.529393 4898 generic.go:356] "Generic (PLEG): container finished" podID="6c89e2d9-6024-479f-8022-6bdda00c9c02" containerID="3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a" exitCode=0 Nov 03 23:00:06.529902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.529442 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" Nov 03 23:00:06.529902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.529479 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" event={"ID":"6c89e2d9-6024-479f-8022-6bdda00c9c02","Type":"ContainerDied","Data":"3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a"} Nov 03 23:00:06.529902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.529557 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc" event={"ID":"6c89e2d9-6024-479f-8022-6bdda00c9c02","Type":"ContainerDied","Data":"c2dc45bee9de971dfbbe002a3213e716c89af76a674c02a56a595bb7d10a67e8"} Nov 03 23:00:06.529902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.529579 4898 scope.go:122] "RemoveContainer" containerID="3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a" Nov 03 23:00:06.540898 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.540712 4898 scope.go:122] "RemoveContainer" containerID="3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a" Nov 03 23:00:06.541141 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:06.541099 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a\": container with ID starting with 3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a not found: ID does not exist" containerID="3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a" Nov 03 23:00:06.541227 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.541146 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a"} err="failed to get container status \"3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a\": rpc error: code = NotFound desc = could not find container \"3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a\": container with ID starting with 3679ce8e9385aff25e20bc102a20f16a0e7b3fda4214f81dd148514ecfc8a61a not found: ID does not exist" Nov 03 23:00:06.697469 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.697431 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc"] Nov 03 23:00:06.706330 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.706296 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9674f5-b6b68d5f4-gnfqc"] Nov 03 23:00:06.907769 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.907735 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ed0f01-5467-4f3c-b635-26f79ef2bd59" path="/var/lib/kubelet/pods/42ed0f01-5467-4f3c-b635-26f79ef2bd59/volumes" Nov 03 23:00:06.908265 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:06.908248 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c89e2d9-6024-479f-8022-6bdda00c9c02" path="/var/lib/kubelet/pods/6c89e2d9-6024-479f-8022-6bdda00c9c02/volumes" Nov 03 23:00:07.747276 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.747243 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr"] Nov 03 23:00:07.747860 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.747832 4898 cpu_manager.go:414] "containerMap: removing container" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" Nov 03 23:00:07.747860 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.747859 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" Nov 03 23:00:07.747989 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.747876 4898 cpu_manager.go:414] "containerMap: removing container" podUID="6c89e2d9-6024-479f-8022-6bdda00c9c02" containerName="maas-api" Nov 03 23:00:07.747989 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.747890 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c89e2d9-6024-479f-8022-6bdda00c9c02" containerName="maas-api" Nov 03 23:00:07.747989 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.747920 4898 cpu_manager.go:414] "containerMap: removing container" podUID="42ed0f01-5467-4f3c-b635-26f79ef2bd59" containerName="maas-api" Nov 03 23:00:07.747989 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.747929 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ed0f01-5467-4f3c-b635-26f79ef2bd59" containerName="maas-api" Nov 03 23:00:07.748147 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.748060 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" Nov 03 23:00:07.748147 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.748072 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" Nov 03 23:00:07.748147 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.748080 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c89e2d9-6024-479f-8022-6bdda00c9c02" containerName="maas-api" Nov 03 23:00:07.748147 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.748088 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ed0f01-5467-4f3c-b635-26f79ef2bd59" containerName="maas-api" Nov 03 23:00:07.753124 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.753101 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:07.811665 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.811630 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr"] Nov 03 23:00:07.842631 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.825423 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:07.853834 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.853757 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:00:07.899310 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.899270 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:07.899499 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.899378 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:07.899499 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:07.899466 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4h4\" (UniqueName: \"kubernetes.io/projected/cccc36dc-6067-4498-a005-2a256c0069c2-kube-api-access-4b4h4\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.000924 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.000820 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4h4\" (UniqueName: \"kubernetes.io/projected/cccc36dc-6067-4498-a005-2a256c0069c2-kube-api-access-4b4h4\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.001607 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.001058 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.001607 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.001133 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.001721 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.001689 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.003616 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.003593 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.062536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.062499 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4h4\" (UniqueName: \"kubernetes.io/projected/cccc36dc-6067-4498-a005-2a256c0069c2-kube-api-access-4b4h4\") pod \"maas-api-revoke-sub-5a47cc74-dwzqr\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.067982 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.067953 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:08.445360 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.445154 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr"] Nov 03 23:00:08.446543 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:00:08.446516 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccc36dc_6067_4498_a005_2a256c0069c2.slice/crio-b8f2c479bb9013adf28ed0c401f12d041facc2760748774484a7712bf3495c8b WatchSource:0}: Error finding container b8f2c479bb9013adf28ed0c401f12d041facc2760748774484a7712bf3495c8b: Status 404 returned error can't find the container with id b8f2c479bb9013adf28ed0c401f12d041facc2760748774484a7712bf3495c8b Nov 03 23:00:08.546527 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:08.546493 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" event={"ID":"cccc36dc-6067-4498-a005-2a256c0069c2","Type":"ContainerStarted","Data":"b8f2c479bb9013adf28ed0c401f12d041facc2760748774484a7712bf3495c8b"} Nov 03 23:00:09.554783 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:09.554693 4898 generic.go:356] "Generic (PLEG): container finished" podID="cccc36dc-6067-4498-a005-2a256c0069c2" containerID="79942930f6f4181be66bca4293cdb69fe0a123d89c14c51b8953367ddda52b89" exitCode=0 Nov 03 23:00:09.554783 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:09.554737 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" event={"ID":"cccc36dc-6067-4498-a005-2a256c0069c2","Type":"ContainerDied","Data":"79942930f6f4181be66bca4293cdb69fe0a123d89c14c51b8953367ddda52b89"} Nov 03 23:00:10.734162 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.734134 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:10.839605 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.839569 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-xks-ca\") pod \"cccc36dc-6067-4498-a005-2a256c0069c2\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " Nov 03 23:00:10.839831 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.839636 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccc36dc-6067-4498-a005-2a256c0069c2-kube-api-access-4b4h4\" (UniqueName: \"kubernetes.io/projected/cccc36dc-6067-4498-a005-2a256c0069c2-kube-api-access-4b4h4\") pod \"cccc36dc-6067-4498-a005-2a256c0069c2\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " Nov 03 23:00:10.839831 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.839748 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-openshift-ca\") pod \"cccc36dc-6067-4498-a005-2a256c0069c2\" (UID: \"cccc36dc-6067-4498-a005-2a256c0069c2\") " Nov 03 23:00:10.840147 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.840113 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-openshift-ca" pod "cccc36dc-6067-4498-a005-2a256c0069c2" (UID: "cccc36dc-6067-4498-a005-2a256c0069c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:10.840347 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.840329 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-openshift-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:10.841878 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.841850 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-xks-ca" pod "cccc36dc-6067-4498-a005-2a256c0069c2" (UID: "cccc36dc-6067-4498-a005-2a256c0069c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:10.841982 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.841889 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccc36dc-6067-4498-a005-2a256c0069c2-kube-api-access-4b4h4" pod "cccc36dc-6067-4498-a005-2a256c0069c2" (UID: "cccc36dc-6067-4498-a005-2a256c0069c2"). InnerVolumeSpecName "kube-api-access-4b4h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:10.941693 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.941655 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccc36dc-6067-4498-a005-2a256c0069c2-maas-api-xks-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:10.941693 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:10.941687 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4h4\" (UniqueName: \"kubernetes.io/projected/cccc36dc-6067-4498-a005-2a256c0069c2-kube-api-access-4b4h4\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:11.408514 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:11.408162 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc"] Nov 03 23:00:11.408824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:11.408754 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" containerID="cri-o://a27e8f6d6863c79cdc1eddae63dcd0931ff10216336e2fd7258d5f4c0449556c" gracePeriod=30 Nov 03 23:00:11.414068 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:11.414029 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght"] Nov 03 23:00:11.414353 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:11.414323 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" containerID="cri-o://2ea2e045aeeed925eadb437115c413afad7f5cc34e22dfa28514eb8e02af652d" gracePeriod=30 Nov 03 23:00:11.588821 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:11.587984 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" event={"ID":"cccc36dc-6067-4498-a005-2a256c0069c2","Type":"ContainerDied","Data":"b8f2c479bb9013adf28ed0c401f12d041facc2760748774484a7712bf3495c8b"} Nov 03 23:00:11.588821 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:11.588051 4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f2c479bb9013adf28ed0c401f12d041facc2760748774484a7712bf3495c8b" Nov 03 23:00:11.588821 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:11.588120 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr" Nov 03 23:00:14.821269 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.821221 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7"] Nov 03 23:00:14.821896 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.821879 4898 cpu_manager.go:414] "containerMap: removing container" podUID="cccc36dc-6067-4498-a005-2a256c0069c2" containerName="revoke-keys" Nov 03 23:00:14.821953 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.821897 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cccc36dc-6067-4498-a005-2a256c0069c2" containerName="revoke-keys" Nov 03 23:00:14.821953 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.821911 4898 cpu_manager.go:414] "containerMap: removing container" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" Nov 03 23:00:14.821953 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.821916 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b55968-7e4c-4508-b0b4-a64ad6eeb74d" containerName="istio-proxy" Nov 03 23:00:14.822051 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.822038 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cccc36dc-6067-4498-a005-2a256c0069c2" containerName="revoke-keys" Nov 03 23:00:14.825709 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.825695 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:14.881703 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.881665 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7"] Nov 03 23:00:14.920818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.920763 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:14.921016 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.920911 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:14.921131 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.921107 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkz5r\" (UniqueName: \"kubernetes.io/projected/056ef2c0-69a6-40ec-9f63-67f3bca670ce-kube-api-access-xkz5r\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:14.927014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.926991 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:14.927130 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:14.927063 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:00:15.022309 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.022269 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:15.022482 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.022356 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:15.022567 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.022497 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkz5r\" (UniqueName: \"kubernetes.io/projected/056ef2c0-69a6-40ec-9f63-67f3bca670ce-kube-api-access-xkz5r\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:15.023110 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.023080 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:15.024684 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.024653 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:15.083871 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.083782 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkz5r\" (UniqueName: \"kubernetes.io/projected/056ef2c0-69a6-40ec-9f63-67f3bca670ce-kube-api-access-xkz5r\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:15.140706 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.140666 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:15.519106 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.519024 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7"] Nov 03 23:00:15.524509 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:00:15.524470 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056ef2c0_69a6_40ec_9f63_67f3bca670ce.slice/crio-41465e041b7fb8683b5d261575304001d90dfc19d5bdcfe81d952c334a0c9a65 WatchSource:0}: Error finding container 41465e041b7fb8683b5d261575304001d90dfc19d5bdcfe81d952c334a0c9a65: Status 404 returned error can't find the container with id 41465e041b7fb8683b5d261575304001d90dfc19d5bdcfe81d952c334a0c9a65 Nov 03 23:00:15.623784 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:15.623748 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" event={"ID":"056ef2c0-69a6-40ec-9f63-67f3bca670ce","Type":"ContainerStarted","Data":"41465e041b7fb8683b5d261575304001d90dfc19d5bdcfe81d952c334a0c9a65"} Nov 03 23:00:16.173628 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.173535 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx"] Nov 03 23:00:16.174131 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.173820 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" containerID="cri-o://26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401" gracePeriod=30 Nov 03 23:00:16.642824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.635829 4898 generic.go:356] "Generic (PLEG): container finished" podID="056ef2c0-69a6-40ec-9f63-67f3bca670ce" containerID="efceb607689d13d6c7bf6d8c03332c27a6a02a5f2ed289c04bc80fa446683c04" exitCode=0 Nov 03 23:00:16.642824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.636023 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" event={"ID":"056ef2c0-69a6-40ec-9f63-67f3bca670ce","Type":"ContainerDied","Data":"efceb607689d13d6c7bf6d8c03332c27a6a02a5f2ed289c04bc80fa446683c04"} Nov 03 23:00:16.642824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.639387 4898 generic.go:356] "Generic (PLEG): container finished" podID="e576686a-090f-4836-b6e3-31024fc18440" containerID="2ea2e045aeeed925eadb437115c413afad7f5cc34e22dfa28514eb8e02af652d" exitCode=0 Nov 03 23:00:16.642824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.639499 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" event={"ID":"e576686a-090f-4836-b6e3-31024fc18440","Type":"ContainerDied","Data":"2ea2e045aeeed925eadb437115c413afad7f5cc34e22dfa28514eb8e02af652d"} Nov 03 23:00:16.642824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.640824 4898 generic.go:356] "Generic (PLEG): container finished" podID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerID="a27e8f6d6863c79cdc1eddae63dcd0931ff10216336e2fd7258d5f4c0449556c" exitCode=0 Nov 03 23:00:16.642824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.640858 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" event={"ID":"a45923fe-4cf8-4adb-93c8-b84d41155806","Type":"ContainerDied","Data":"a27e8f6d6863c79cdc1eddae63dcd0931ff10216336e2fd7258d5f4c0449556c"} Nov 03 23:00:16.726349 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.726290 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 23:00:16.729689 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.729668 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 23:00:16.855891 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.855853 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-podinfo\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856043 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.855926 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45923fe-4cf8-4adb-93c8-b84d41155806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45923fe-4cf8-4adb-93c8-b84d41155806-istiod-ca-cert\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856043 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.855955 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-certs\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856043 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.855972 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-certs\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856043 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856019 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-token\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-token\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856043 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856038 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-kube-api-access-8g4sn\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-kube-api-access-8g4sn\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856074 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-kube-api-access-hktzz\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-kube-api-access-hktzz\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856105 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e576686a-090f-4836-b6e3-31024fc18440-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e576686a-090f-4836-b6e3-31024fc18440-istio-podinfo\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856171 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-credential-socket\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856201 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-socket\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856223 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-data\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856253 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-envoy\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856254 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-certs" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.856319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856292 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-credential-socket\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856374 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e576686a-090f-4836-b6e3-31024fc18440-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e576686a-090f-4836-b6e3-31024fc18440-istiod-ca-cert\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856384 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-certs" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.856742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856416 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-envoy\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856462 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-istio-token\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-istio-token\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.856742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856505 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-credential-socket" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.856742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856518 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-socket\") pod \"a45923fe-4cf8-4adb-93c8-b84d41155806\" (UID: \"a45923fe-4cf8-4adb-93c8-b84d41155806\") " Nov 03 23:00:16.856742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856568 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-data\") pod \"e576686a-090f-4836-b6e3-31024fc18440\" (UID: \"e576686a-090f-4836-b6e3-31024fc18440\") " Nov 03 23:00:16.857255 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.857229 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.857330 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.857262 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.857330 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.857280 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.857691 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856433 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45923fe-4cf8-4adb-93c8-b84d41155806-istiod-ca-cert" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:16.857802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856715 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-socket" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.857802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856759 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e576686a-090f-4836-b6e3-31024fc18440-istiod-ca-cert" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:16.857802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.856931 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-credential-socket" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.857802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.857031 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-data" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.857802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.857504 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-data" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.857802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.857667 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-socket" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.858946 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.858871 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-podinfo" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:16.858946 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.858935 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-kube-api-access-hktzz" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "kube-api-access-hktzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.859149 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.859082 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e576686a-090f-4836-b6e3-31024fc18440-istio-podinfo" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:16.859320 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.859273 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-token" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.859320 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.859301 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-istio-token" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.859462 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.859339 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-envoy" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.859560 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.859539 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-kube-api-access-8g4sn" pod "a45923fe-4cf8-4adb-93c8-b84d41155806" (UID: "a45923fe-4cf8-4adb-93c8-b84d41155806"). InnerVolumeSpecName "kube-api-access-8g4sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.859765 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.859753 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-envoy" pod "e576686a-090f-4836-b6e3-31024fc18440" (UID: "e576686a-090f-4836-b6e3-31024fc18440"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959317 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktzz\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-kube-api-access-hktzz\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959358 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e576686a-090f-4836-b6e3-31024fc18440-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959374 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959389 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959404 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959420 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959435 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e576686a-090f-4836-b6e3-31024fc18440-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959448 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959462 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e576686a-090f-4836-b6e3-31024fc18440-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959476 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45923fe-4cf8-4adb-93c8-b84d41155806-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959490 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e576686a-090f-4836-b6e3-31024fc18440-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959503 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959516 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45923fe-4cf8-4adb-93c8-b84d41155806-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959530 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.963843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:16.959546 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g4sn\" (UniqueName: \"kubernetes.io/projected/a45923fe-4cf8-4adb-93c8-b84d41155806-kube-api-access-8g4sn\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:17.651379 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.651337 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" event={"ID":"e576686a-090f-4836-b6e3-31024fc18440","Type":"ContainerDied","Data":"2871cf40561b9bae0a52dd47442d77eb45bf9a3f9a5a4765faa3394088c905f3"} Nov 03 23:00:17.651918 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.651418 4898 scope.go:122] "RemoveContainer" containerID="2ea2e045aeeed925eadb437115c413afad7f5cc34e22dfa28514eb8e02af652d" Nov 03 23:00:17.651918 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.651579 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght" Nov 03 23:00:17.654760 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.654736 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" Nov 03 23:00:17.655351 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.655324 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc" event={"ID":"a45923fe-4cf8-4adb-93c8-b84d41155806","Type":"ContainerDied","Data":"873bbc43f89de42f5a19361b62a764367e8020b6f657b5908b895958fec463a8"} Nov 03 23:00:17.669898 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.669855 4898 scope.go:122] "RemoveContainer" containerID="a27e8f6d6863c79cdc1eddae63dcd0931ff10216336e2fd7258d5f4c0449556c" Nov 03 23:00:17.821059 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.821028 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc"] Nov 03 23:00:17.828191 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.828164 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:17.871826 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.871772 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-zp9gc"] Nov 03 23:00:17.874049 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.874015 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-openshift-ca\") pod \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " Nov 03 23:00:17.874204 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.874173 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056ef2c0-69a6-40ec-9f63-67f3bca670ce-kube-api-access-xkz5r\" (UniqueName: \"kubernetes.io/projected/056ef2c0-69a6-40ec-9f63-67f3bca670ce-kube-api-access-xkz5r\") pod \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " Nov 03 23:00:17.874269 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.874216 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-xks-ca\") pod \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\" (UID: \"056ef2c0-69a6-40ec-9f63-67f3bca670ce\") " Nov 03 23:00:17.874411 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.874380 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-openshift-ca" pod "056ef2c0-69a6-40ec-9f63-67f3bca670ce" (UID: "056ef2c0-69a6-40ec-9f63-67f3bca670ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:17.874615 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.874598 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-openshift-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:17.876414 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.876386 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-xks-ca" pod "056ef2c0-69a6-40ec-9f63-67f3bca670ce" (UID: "056ef2c0-69a6-40ec-9f63-67f3bca670ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:17.876599 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.876579 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056ef2c0-69a6-40ec-9f63-67f3bca670ce-kube-api-access-xkz5r" pod "056ef2c0-69a6-40ec-9f63-67f3bca670ce" (UID: "056ef2c0-69a6-40ec-9f63-67f3bca670ce"). InnerVolumeSpecName "kube-api-access-xkz5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:17.976168 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.976083 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkz5r\" (UniqueName: \"kubernetes.io/projected/056ef2c0-69a6-40ec-9f63-67f3bca670ce-kube-api-access-xkz5r\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:17.976168 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.976124 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056ef2c0-69a6-40ec-9f63-67f3bca670ce-maas-api-xks-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:17.980972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.980933 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght"] Nov 03 23:00:17.985784 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:17.985758 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-6zght"] Nov 03 23:00:18.666721 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:18.666683 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" Nov 03 23:00:18.667249 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:18.666715 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7" event={"ID":"056ef2c0-69a6-40ec-9f63-67f3bca670ce","Type":"ContainerDied","Data":"41465e041b7fb8683b5d261575304001d90dfc19d5bdcfe81d952c334a0c9a65"} Nov 03 23:00:18.667249 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:18.667140 4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41465e041b7fb8683b5d261575304001d90dfc19d5bdcfe81d952c334a0c9a65" Nov 03 23:00:18.906908 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:18.906875 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" path="/var/lib/kubelet/pods/a45923fe-4cf8-4adb-93c8-b84d41155806/volumes" Nov 03 23:00:18.907370 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:18.907356 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e576686a-090f-4836-b6e3-31024fc18440" path="/var/lib/kubelet/pods/e576686a-090f-4836-b6e3-31024fc18440/volumes" Nov 03 23:00:19.835578 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:19.835536 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:00:21.459569 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.459506 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 23:00:21.520583 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520553 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-envoy\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.520776 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520610 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-token\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-token\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.520776 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520665 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istiod-ca-cert\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.520776 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520692 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-socket\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.520776 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520723 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-data\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.521017 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520783 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-certs\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.521017 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520891 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-kube-api-access-z56r5\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-kube-api-access-z56r5\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.521017 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520944 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-podinfo\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.521017 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.520976 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-credential-socket\") pod \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\" (UID: \"661c0a78-c7ec-4875-a439-fafe63b9c2d2\") " Nov 03 23:00:21.521220 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521147 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-socket" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:21.521220 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521166 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-data" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:21.521445 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521242 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-certs" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:21.521445 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521371 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-credential-socket" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:21.521598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521571 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.521737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521601 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.521737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521618 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.521737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521631 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.521737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.521631 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istiod-ca-cert" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.523018 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.522987 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-envoy" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:21.523134 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.523107 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-token" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:21.523251 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.523231 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-podinfo" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:21.523366 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.523351 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-kube-api-access-z56r5" pod "661c0a78-c7ec-4875-a439-fafe63b9c2d2" (UID: "661c0a78-c7ec-4875-a439-fafe63b9c2d2"). InnerVolumeSpecName "kube-api-access-z56r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:21.623358 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.623312 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.623358 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.623345 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.623358 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.623354 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.623358 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.623364 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z56r5\" (UniqueName: \"kubernetes.io/projected/661c0a78-c7ec-4875-a439-fafe63b9c2d2-kube-api-access-z56r5\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.623723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.623375 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661c0a78-c7ec-4875-a439-fafe63b9c2d2-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.690573 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.690533 4898 generic.go:356] "Generic (PLEG): container finished" podID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerID="26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401" exitCode=0 Nov 03 23:00:21.690745 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.690587 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" Nov 03 23:00:21.690745 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.690615 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" event={"ID":"661c0a78-c7ec-4875-a439-fafe63b9c2d2","Type":"ContainerDied","Data":"26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401"} Nov 03 23:00:21.690745 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.690672 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx" event={"ID":"661c0a78-c7ec-4875-a439-fafe63b9c2d2","Type":"ContainerDied","Data":"1d23ad4197f961b39e381691f28976678868f1b6341e3a6e26a48f47ced7c06d"} Nov 03 23:00:21.690745 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.690694 4898 scope.go:122] "RemoveContainer" containerID="26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401" Nov 03 23:00:21.701756 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.701734 4898 scope.go:122] "RemoveContainer" containerID="958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50" Nov 03 23:00:21.710853 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.710834 4898 scope.go:122] "RemoveContainer" containerID="26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401" Nov 03 23:00:21.711106 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:21.711086 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401\": container with ID starting with 26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401 not found: ID does not exist" containerID="26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401" Nov 03 23:00:21.711174 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.711116 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401"} err="failed to get container status \"26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401\": rpc error: code = NotFound desc = could not find container \"26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401\": container with ID starting with 26221815605f26f056a4249dfab4530324d585da7a15ace70e5458b73f4be401 not found: ID does not exist" Nov 03 23:00:21.711174 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.711134 4898 scope.go:122] "RemoveContainer" containerID="958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50" Nov 03 23:00:21.711316 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:21.711298 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50\": container with ID starting with 958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50 not found: ID does not exist" containerID="958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50" Nov 03 23:00:21.711352 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.711324 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50"} err="failed to get container status \"958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50\": rpc error: code = NotFound desc = could not find container \"958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50\": container with ID starting with 958c499b3ccc4f05234df6aecf4e44d48029bd7ea7d8dd23620f9fd7d8140e50 not found: ID does not exist" Nov 03 23:00:21.858417 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.858378 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx"] Nov 03 23:00:21.861756 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:21.861724 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9674f5-openshift-default-7b45c9f895-w26vx"] Nov 03 23:00:22.907005 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:22.906958 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" path="/var/lib/kubelet/pods/661c0a78-c7ec-4875-a439-fafe63b9c2d2/volumes" Nov 03 23:00:28.145068 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.145033 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt"] Nov 03 23:00:28.145579 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.145238 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" podUID="79c9ca07-ee0b-474f-9954-087baaf18359" containerName="maas-api" containerID="cri-o://f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf" gracePeriod=30 Nov 03 23:00:28.464198 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.464162 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 23:00:28.621476 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.621442 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c9ca07-ee0b-474f-9954-087baaf18359-kube-api-access-g6wxk\" (UniqueName: \"kubernetes.io/projected/79c9ca07-ee0b-474f-9954-087baaf18359-kube-api-access-g6wxk\") pod \"79c9ca07-ee0b-474f-9954-087baaf18359\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " Nov 03 23:00:28.621730 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.621572 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c9ca07-ee0b-474f-9954-087baaf18359-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c9ca07-ee0b-474f-9954-087baaf18359-openshift-service-ca\") pod \"79c9ca07-ee0b-474f-9954-087baaf18359\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " Nov 03 23:00:28.621730 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.621632 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-maas-api-tls\") pod \"79c9ca07-ee0b-474f-9954-087baaf18359\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " Nov 03 23:00:28.621939 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.621759 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-metrics-certs\") pod \"79c9ca07-ee0b-474f-9954-087baaf18359\" (UID: \"79c9ca07-ee0b-474f-9954-087baaf18359\") " Nov 03 23:00:28.624781 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.624744 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c9ca07-ee0b-474f-9954-087baaf18359-openshift-service-ca" pod "79c9ca07-ee0b-474f-9954-087baaf18359" (UID: "79c9ca07-ee0b-474f-9954-087baaf18359"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:28.625350 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.625311 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c9ca07-ee0b-474f-9954-087baaf18359-kube-api-access-g6wxk" pod "79c9ca07-ee0b-474f-9954-087baaf18359" (UID: "79c9ca07-ee0b-474f-9954-087baaf18359"). InnerVolumeSpecName "kube-api-access-g6wxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:28.627196 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.626559 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-metrics-certs" pod "79c9ca07-ee0b-474f-9954-087baaf18359" (UID: "79c9ca07-ee0b-474f-9954-087baaf18359"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:28.637010 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.636972 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-maas-api-tls" pod "79c9ca07-ee0b-474f-9954-087baaf18359" (UID: "79c9ca07-ee0b-474f-9954-087baaf18359"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:28.723095 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.723006 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:28.723095 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.723038 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6wxk\" (UniqueName: \"kubernetes.io/projected/79c9ca07-ee0b-474f-9954-087baaf18359-kube-api-access-g6wxk\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:28.723095 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.723054 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c9ca07-ee0b-474f-9954-087baaf18359-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:28.723095 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.723071 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c9ca07-ee0b-474f-9954-087baaf18359-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:28.744859 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.744813 4898 generic.go:356] "Generic (PLEG): container finished" podID="79c9ca07-ee0b-474f-9954-087baaf18359" containerID="f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf" exitCode=0 Nov 03 23:00:28.745014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.744915 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" Nov 03 23:00:28.745014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.744984 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" event={"ID":"79c9ca07-ee0b-474f-9954-087baaf18359","Type":"ContainerDied","Data":"f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf"} Nov 03 23:00:28.745144 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.745034 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt" event={"ID":"79c9ca07-ee0b-474f-9954-087baaf18359","Type":"ContainerDied","Data":"9db7ff2eb7589d2df63e86629cdb000b84a0c59fc9afd983abdfbbff31cbcbfa"} Nov 03 23:00:28.745144 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.745061 4898 scope.go:122] "RemoveContainer" containerID="f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf" Nov 03 23:00:28.757272 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.757245 4898 scope.go:122] "RemoveContainer" containerID="f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf" Nov 03 23:00:28.757570 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:28.757551 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf\": container with ID starting with f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf not found: ID does not exist" containerID="f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf" Nov 03 23:00:28.757619 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.757579 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf"} err="failed to get container status \"f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf\": rpc error: code = NotFound desc = could not find container \"f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf\": container with ID starting with f2006f9b16667dbbe8192fda37b23a34a4120dad49b102590523f0f93db82abf not found: ID does not exist" Nov 03 23:00:28.914305 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.914272 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt"] Nov 03 23:00:28.919001 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:28.918974 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-65cac0-65c4b55bc8-kqqqt"] Nov 03 23:00:30.906831 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:30.906766 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c9ca07-ee0b-474f-9954-087baaf18359" path="/var/lib/kubelet/pods/79c9ca07-ee0b-474f-9954-087baaf18359/volumes" Nov 03 23:00:40.119273 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119238 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn"] Nov 03 23:00:40.119854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119839 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" Nov 03 23:00:40.119923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119854 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" Nov 03 23:00:40.119923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119868 4898 cpu_manager.go:414] "containerMap: removing container" podUID="056ef2c0-69a6-40ec-9f63-67f3bca670ce" containerName="revoke-keys" Nov 03 23:00:40.119923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119873 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056ef2c0-69a6-40ec-9f63-67f3bca670ce" containerName="revoke-keys" Nov 03 23:00:40.119923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119882 4898 cpu_manager.go:414] "containerMap: removing container" podUID="79c9ca07-ee0b-474f-9954-087baaf18359" containerName="maas-api" Nov 03 23:00:40.119923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119896 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c9ca07-ee0b-474f-9954-087baaf18359" containerName="maas-api" Nov 03 23:00:40.119923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119909 4898 cpu_manager.go:414] "containerMap: removing container" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" Nov 03 23:00:40.119923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119914 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119928 4898 cpu_manager.go:414] "containerMap: removing container" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119933 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119954 4898 cpu_manager.go:414] "containerMap: removing container" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.119958 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.120038 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c9ca07-ee0b-474f-9954-087baaf18359" containerName="maas-api" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.120049 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.120056 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="661c0a78-c7ec-4875-a439-fafe63b9c2d2" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.120065 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e576686a-090f-4836-b6e3-31024fc18440" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.120075 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45923fe-4cf8-4adb-93c8-b84d41155806" containerName="istio-proxy" Nov 03 23:00:40.120142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.120083 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="056ef2c0-69a6-40ec-9f63-67f3bca670ce" containerName="revoke-keys" Nov 03 23:00:40.126085 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.126064 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.178855 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.178811 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2cea18-serving-cert\"" Nov 03 23:00:40.189611 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.189575 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn"] Nov 03 23:00:40.203652 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.203621 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-metrics-certs\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.203823 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.203739 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf69038-2c53-4767-ab1e-a0447f0af9dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.203933 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.203904 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.204075 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.204057 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjff\" (UniqueName: \"kubernetes.io/projected/2bf69038-2c53-4767-ab1e-a0447f0af9dd-kube-api-access-gmjff\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.304756 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.304722 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf69038-2c53-4767-ab1e-a0447f0af9dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.304957 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.304815 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.304957 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.304888 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjff\" (UniqueName: \"kubernetes.io/projected/2bf69038-2c53-4767-ab1e-a0447f0af9dd-kube-api-access-gmjff\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.304957 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.304922 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-metrics-certs\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.305129 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:40.304970 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-serving-cert: secret "maas-api-e2e-inf-b-2cea18-serving-cert" not found Nov 03 23:00:40.305129 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:40.305046 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls podName:2bf69038-2c53-4767-ab1e-a0447f0af9dd nodeName:}" failed. No retries permitted until 2026-11-03 23:00:40.805024791 +0000 UTC m=+2284.017793842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls") pod "maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" (UID: "2bf69038-2c53-4767-ab1e-a0447f0af9dd") : secret "maas-api-e2e-inf-b-2cea18-serving-cert" not found Nov 03 23:00:40.305473 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.305452 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf69038-2c53-4767-ab1e-a0447f0af9dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.307372 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.307350 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-metrics-certs\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.366774 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.366742 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjff\" (UniqueName: \"kubernetes.io/projected/2bf69038-2c53-4767-ab1e-a0447f0af9dd-kube-api-access-gmjff\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.818802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.818755 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:40.821438 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:40.821406 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:41.039082 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:41.039043 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:41.426164 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:41.426133 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn"] Nov 03 23:00:41.427738 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:00:41.427710 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf69038_2c53_4767_ab1e_a0447f0af9dd.slice/crio-dd29a09b36129b9845e0bf794ec405e2bcdaa6473b2d7ddce76aa744f588bd0d WatchSource:0}: Error finding container dd29a09b36129b9845e0bf794ec405e2bcdaa6473b2d7ddce76aa744f588bd0d: Status 404 returned error can't find the container with id dd29a09b36129b9845e0bf794ec405e2bcdaa6473b2d7ddce76aa744f588bd0d Nov 03 23:00:41.429369 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:41.429347 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:00:41.827266 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:41.827232 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" event={"ID":"2bf69038-2c53-4767-ab1e-a0447f0af9dd","Type":"ContainerStarted","Data":"dd29a09b36129b9845e0bf794ec405e2bcdaa6473b2d7ddce76aa744f588bd0d"} Nov 03 23:00:42.835210 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:42.835173 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" event={"ID":"2bf69038-2c53-4767-ab1e-a0447f0af9dd","Type":"ContainerStarted","Data":"10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb"} Nov 03 23:00:42.835712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:42.835239 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:42.956064 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:42.956007 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" podStartSLOduration=2.055914497 podStartE2EDuration="2.955991424s" totalImagesPullingTime="900.076927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:42.952854918 +0000 UTC m=+2286.165623989" watchObservedRunningTime="2026-11-03 23:00:42.955991424 +0000 UTC m=+2286.168760475" Nov 03 23:00:48.850492 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:48.850461 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:00:49.835431 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:00:49.835391 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:00:59.069527 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.069495 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2"] Nov 03 23:00:59.069977 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.069708 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" podUID="c7fb8b42-5285-4695-9cfa-cb4214b3ba03" containerName="maas-api" containerID="cri-o://0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962" gracePeriod=30 Nov 03 23:00:59.401330 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.401299 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 23:00:59.454818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.445627 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-metrics-certs\") pod \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " Nov 03 23:00:59.456412 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.455191 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-openshift-service-ca\") pod \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " Nov 03 23:00:59.456412 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.455279 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls\") pod \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " Nov 03 23:00:59.456412 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.455364 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-kube-api-access-b96bc\" (UniqueName: \"kubernetes.io/projected/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-kube-api-access-b96bc\") pod \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\" (UID: \"c7fb8b42-5285-4695-9cfa-cb4214b3ba03\") " Nov 03 23:00:59.467018 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.466418 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-openshift-service-ca" pod "c7fb8b42-5285-4695-9cfa-cb4214b3ba03" (UID: "c7fb8b42-5285-4695-9cfa-cb4214b3ba03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:59.472449 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.472389 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-metrics-certs" pod "c7fb8b42-5285-4695-9cfa-cb4214b3ba03" (UID: "c7fb8b42-5285-4695-9cfa-cb4214b3ba03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:59.485720 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.482445 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-kube-api-access-b96bc" pod "c7fb8b42-5285-4695-9cfa-cb4214b3ba03" (UID: "c7fb8b42-5285-4695-9cfa-cb4214b3ba03"). InnerVolumeSpecName "kube-api-access-b96bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:59.503809 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.497847 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls" pod "c7fb8b42-5285-4695-9cfa-cb4214b3ba03" (UID: "c7fb8b42-5285-4695-9cfa-cb4214b3ba03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:59.557829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.557498 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:59.557829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.557549 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:59.557829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.557567 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b96bc\" (UniqueName: \"kubernetes.io/projected/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-kube-api-access-b96bc\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:59.557829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.557581 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fb8b42-5285-4695-9cfa-cb4214b3ba03-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:59.998674 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.998639 4898 generic.go:356] "Generic (PLEG): container finished" podID="c7fb8b42-5285-4695-9cfa-cb4214b3ba03" containerID="0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962" exitCode=0 Nov 03 23:00:59.998882 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.998694 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" Nov 03 23:00:59.998882 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.998731 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" event={"ID":"c7fb8b42-5285-4695-9cfa-cb4214b3ba03","Type":"ContainerDied","Data":"0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962"} Nov 03 23:00:59.998882 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.998813 4898 scope.go:122] "RemoveContainer" containerID="0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962" Nov 03 23:00:59.999044 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:00:59.998817 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2" event={"ID":"c7fb8b42-5285-4695-9cfa-cb4214b3ba03","Type":"ContainerDied","Data":"5d2a1c53c7b1534c276fd485671909d0663ff872226e11c0be76c83a56126511"} Nov 03 23:01:00.028659 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:00.027977 4898 scope.go:122] "RemoveContainer" containerID="0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962" Nov 03 23:01:00.028968 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:01:00.028943 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962\": container with ID starting with 0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962 not found: ID does not exist" containerID="0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962" Nov 03 23:01:00.029167 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:00.029125 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962"} err="failed to get container status \"0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962\": rpc error: code = NotFound desc = could not find container \"0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962\": container with ID starting with 0df69c22be78df138864c649e323e1f097830fcc3b9e43d133c7ac743cfbb962 not found: ID does not exist" Nov 03 23:01:00.165304 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:00.165267 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2"] Nov 03 23:01:00.215587 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:00.215545 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6296b-5dd55b7db9-f9bm2"] Nov 03 23:01:00.907848 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:00.907810 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fb8b42-5285-4695-9cfa-cb4214b3ba03" path="/var/lib/kubelet/pods/c7fb8b42-5285-4695-9cfa-cb4214b3ba03/volumes" Nov 03 23:01:08.086505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:08.086261 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x"] Nov 03 23:01:08.087210 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:08.087174 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" containerID="cri-o://3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d" gracePeriod=30 Nov 03 23:01:10.543231 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.543191 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj"] Nov 03 23:01:10.543851 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.543780 4898 cpu_manager.go:414] "containerMap: removing container" podUID="c7fb8b42-5285-4695-9cfa-cb4214b3ba03" containerName="maas-api" Nov 03 23:01:10.543851 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.543803 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fb8b42-5285-4695-9cfa-cb4214b3ba03" containerName="maas-api" Nov 03 23:01:10.543988 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.543897 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fb8b42-5285-4695-9cfa-cb4214b3ba03" containerName="maas-api" Nov 03 23:01:10.547830 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.547805 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.638688 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.638441 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj"] Nov 03 23:01:10.699861 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.690077 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-656ee6-openshift-default-dockercfg-h4vpn\"" Nov 03 23:01:10.737477 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737443 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-certs\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.737650 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737489 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-istio-token\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.737650 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737603 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-credential-socket\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.737650 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737636 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b11632-5041-429c-aabf-23b9123082e2-istio-podinfo\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.737799 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737751 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-envoy\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.737799 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737775 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-socket\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.737897 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737870 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqgm\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-kube-api-access-2hqgm\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.737955 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.737925 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-data\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.738040 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.738025 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b11632-5041-429c-aabf-23b9123082e2-istiod-ca-cert\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.838763 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.838731 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-certs\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.838996 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.838771 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-istio-token\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.838996 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.838844 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-credential-socket\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.838996 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.838872 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b11632-5041-429c-aabf-23b9123082e2-istio-podinfo\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.838996 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.838948 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-envoy\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.838996 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.838973 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-socket\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839262 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839030 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqgm\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-kube-api-access-2hqgm\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839262 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839055 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-data\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839262 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839122 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b11632-5041-429c-aabf-23b9123082e2-istiod-ca-cert\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839262 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839206 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-certs\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839476 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839216 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-credential-socket\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839476 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839461 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-socket\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839840 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839813 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-data\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.839973 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.839936 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b11632-5041-429c-aabf-23b9123082e2-istiod-ca-cert\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.841750 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.841720 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b11632-5041-429c-aabf-23b9123082e2-istio-podinfo\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.841989 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.841969 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-envoy\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.900110 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.900072 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-istio-token\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:10.903090 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:10.903064 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqgm\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-kube-api-access-2hqgm\") pod \"e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:11.180127 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:11.180025 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:11.619476 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:11.619439 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj"] Nov 03 23:01:11.621272 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:01:11.621244 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b11632_5041_429c_aabf_23b9123082e2.slice/crio-0b7c8f7c806c54a2d67da0872261b0ed82bdb1f6fc664c83e3c9a2bd7e0fdca1 WatchSource:0}: Error finding container 0b7c8f7c806c54a2d67da0872261b0ed82bdb1f6fc664c83e3c9a2bd7e0fdca1: Status 404 returned error can't find the container with id 0b7c8f7c806c54a2d67da0872261b0ed82bdb1f6fc664c83e3c9a2bd7e0fdca1 Nov 03 23:01:11.623702 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:11.623666 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:11.623833 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:11.623747 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:11.623833 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:11.623811 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:12.106011 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:12.105972 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" event={"ID":"c9b11632-5041-429c-aabf-23b9123082e2","Type":"ContainerStarted","Data":"b18ee4668f759b81194cac1011c272db830a336d4cd538f79d490185814c8c82"} Nov 03 23:01:12.106229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:12.106039 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" event={"ID":"c9b11632-5041-429c-aabf-23b9123082e2","Type":"ContainerStarted","Data":"0b7c8f7c806c54a2d67da0872261b0ed82bdb1f6fc664c83e3c9a2bd7e0fdca1"} Nov 03 23:01:12.180978 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:12.180933 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:12.187319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:12.187292 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:12.280814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:12.280305 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podStartSLOduration=2.280283383 podStartE2EDuration="2.280283383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:12.279978862 +0000 UTC m=+2315.492747917" watchObservedRunningTime="2026-11-03 23:01:12.280283383 +0000 UTC m=+2315.493052438" Nov 03 23:01:13.116246 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.116213 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:13.117291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.117266 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:01:13.422193 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.422170 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 23:01:13.578169 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.578126 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-certs\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.578773 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.578752 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-data\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.579014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.578994 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-socket\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.579154 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.579141 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-credential-socket\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.579280 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.579266 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-envoy\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.579446 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.579431 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-kube-api-access-zv272\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-kube-api-access-zv272\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.579737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.579714 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2267cf-8669-41cd-acbd-8f02e561853e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2267cf-8669-41cd-acbd-8f02e561853e-istiod-ca-cert\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.580741 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.579955 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-podinfo\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.580741 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.580066 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-token\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-token\") pod \"4d2267cf-8669-41cd-acbd-8f02e561853e\" (UID: \"4d2267cf-8669-41cd-acbd-8f02e561853e\") " Nov 03 23:01:13.582239 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.578577 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-certs" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:13.582239 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.581757 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2267cf-8669-41cd-acbd-8f02e561853e-istiod-ca-cert" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:13.582239 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.582152 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-socket" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:13.582239 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.582216 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-data" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:13.582779 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.582755 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-credential-socket" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:13.583116 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.583089 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-token" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:13.584668 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.584643 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-kube-api-access-zv272" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "kube-api-access-zv272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:13.584969 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.584945 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-podinfo" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:13.590389 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.590359 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-envoy" pod "4d2267cf-8669-41cd-acbd-8f02e561853e" (UID: "4d2267cf-8669-41cd-acbd-8f02e561853e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681305 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681336 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681346 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681355 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv272\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-kube-api-access-zv272\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681365 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2267cf-8669-41cd-acbd-8f02e561853e-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681374 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681382 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681390 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:13.681398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:13.681399 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2267cf-8669-41cd-acbd-8f02e561853e-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:14.123604 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.123568 4898 generic.go:356] "Generic (PLEG): container finished" podID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerID="3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d" exitCode=0 Nov 03 23:01:14.124094 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.123627 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" Nov 03 23:01:14.124094 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.123660 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" event={"ID":"4d2267cf-8669-41cd-acbd-8f02e561853e","Type":"ContainerDied","Data":"3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d"} Nov 03 23:01:14.124094 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.123733 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x" event={"ID":"4d2267cf-8669-41cd-acbd-8f02e561853e","Type":"ContainerDied","Data":"fae6dcca8ce2f08979ffd3fdc9faf1b088e4999e4229e8c851e57752f7401703"} Nov 03 23:01:14.124094 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.123764 4898 scope.go:122] "RemoveContainer" containerID="3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d" Nov 03 23:01:14.135828 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.135778 4898 scope.go:122] "RemoveContainer" containerID="9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04" Nov 03 23:01:14.146831 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.146812 4898 scope.go:122] "RemoveContainer" containerID="3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d" Nov 03 23:01:14.147110 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:01:14.147089 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d\": container with ID starting with 3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d not found: ID does not exist" containerID="3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d" Nov 03 23:01:14.147197 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.147118 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d"} err="failed to get container status \"3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d\": rpc error: code = NotFound desc = could not find container \"3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d\": container with ID starting with 3bf86eb3fa0eeb5b54240c55dd6f210fb1fca67d5d08651700df3af0f1b4d68d not found: ID does not exist" Nov 03 23:01:14.147197 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.147139 4898 scope.go:122] "RemoveContainer" containerID="9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04" Nov 03 23:01:14.147387 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:01:14.147372 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04\": container with ID starting with 9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04 not found: ID does not exist" containerID="9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04" Nov 03 23:01:14.147445 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.147398 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04"} err="failed to get container status \"9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04\": rpc error: code = NotFound desc = could not find container \"9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04\": container with ID starting with 9b59b67e7daeeed2eb14501f7eb8e2dc6830d9854fd566912755621c5ed58c04 not found: ID does not exist" Nov 03 23:01:14.291536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.291504 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x"] Nov 03 23:01:14.294915 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.294890 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d6296b-openshift-default-7f98f9c4d5-p9g4x"] Nov 03 23:01:14.906377 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:14.906341 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" path="/var/lib/kubelet/pods/4d2267cf-8669-41cd-acbd-8f02e561853e/volumes" Nov 03 23:01:19.835033 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:01:19.834960 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:01:23.067813 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:23.066142 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:01:24.672811 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:24.671998 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-std4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:01:24.672811 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:24.672096 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:01:24.832113 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:24.832066 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nr7zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:01:24.832463 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:24.832409 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:01:30.816669 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:30.816626 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7"] Nov 03 23:01:30.825243 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:30.825207 4898 patch_prober.go:29] interesting pod/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:01:30.825506 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:30.825466 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 23:01:30.862864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:30.862775 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-65cac0-9vgg7"] Nov 03 23:01:30.908874 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:30.908833 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056ef2c0-69a6-40ec-9f63-67f3bca670ce" path="/var/lib/kubelet/pods/056ef2c0-69a6-40ec-9f63-67f3bca670ce/volumes" Nov 03 23:01:31.542701 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:31.542662 4898 generic.go:356] "Generic (PLEG): container finished" podID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerID="1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f" exitCode=0 Nov 03 23:01:31.542892 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:31.542693 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" event={"ID":"909a0d33-8e64-42a5-9ed0-67cb0a96a413","Type":"ContainerDied","Data":"1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f"} Nov 03 23:01:31.543204 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:31.543188 4898 scope.go:122] "RemoveContainer" containerID="1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f" Nov 03 23:01:31.544782 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:31.544752 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:31.544923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:31.544846 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:31.544923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:31.544884 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:32.571743 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:32.571709 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" event={"ID":"909a0d33-8e64-42a5-9ed0-67cb0a96a413","Type":"ContainerStarted","Data":"c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194"} Nov 03 23:01:33.390452 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:33.390410 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 23:01:33.390652 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:33.390626 4898 patch_prober.go:29] interesting pod/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:01:33.390699 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:33.390676 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 23:01:34.390894 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:34.390856 4898 patch_prober.go:29] interesting pod/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:01:34.391293 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:34.390910 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 23:01:35.391185 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:35.391147 4898 patch_prober.go:29] interesting pod/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:01:35.391607 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:35.391199 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 23:01:36.391393 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:36.391350 4898 patch_prober.go:29] interesting pod/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:01:36.391820 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:36.391428 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 23:01:37.391303 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:37.391263 4898 patch_prober.go:29] interesting pod/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:01:37.391498 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:37.391320 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 23:01:38.396033 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:38.396001 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 23:01:38.396514 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:38.396415 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 23:01:38.397166 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:38.397148 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 23:01:46.465614 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:01:46.465568 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6"] Nov 03 23:01:49.835846 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:01:49.835777 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:02:01.486592 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.486554 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" containerID="cri-o://7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453" gracePeriod=45 Nov 03 23:02:01.783179 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.783144 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 23:02:01.902441 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902405 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tmp-dir\") pod \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " Nov 03 23:02:01.902647 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902457 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-home\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-home\") pod \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " Nov 03 23:02:01.902647 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902576 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tls-certs\") pod \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " Nov 03 23:02:01.902647 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902624 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fd0e1c-3bb7-4404-be60-6992b66de9ee-kube-api-access-nvp4r\" (UniqueName: \"kubernetes.io/projected/95fd0e1c-3bb7-4404-be60-6992b66de9ee-kube-api-access-nvp4r\") pod \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " Nov 03 23:02:01.902864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902701 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-model-cache\") pod \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " Nov 03 23:02:01.902864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902747 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-home" pod "95fd0e1c-3bb7-4404-be60-6992b66de9ee" (UID: "95fd0e1c-3bb7-4404-be60-6992b66de9ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:01.902864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902763 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tmp-dir" pod "95fd0e1c-3bb7-4404-be60-6992b66de9ee" (UID: "95fd0e1c-3bb7-4404-be60-6992b66de9ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:01.902864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.902831 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-dshm\") pod \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\" (UID: \"95fd0e1c-3bb7-4404-be60-6992b66de9ee\") " Nov 03 23:02:01.903068 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.903004 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-model-cache" pod "95fd0e1c-3bb7-4404-be60-6992b66de9ee" (UID: "95fd0e1c-3bb7-4404-be60-6992b66de9ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:01.903305 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.903284 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:01.903416 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.903308 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:01.903416 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.903322 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:01.905103 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.905076 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fd0e1c-3bb7-4404-be60-6992b66de9ee-kube-api-access-nvp4r" pod "95fd0e1c-3bb7-4404-be60-6992b66de9ee" (UID: "95fd0e1c-3bb7-4404-be60-6992b66de9ee"). InnerVolumeSpecName "kube-api-access-nvp4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:01.905329 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.905312 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-dshm" pod "95fd0e1c-3bb7-4404-be60-6992b66de9ee" (UID: "95fd0e1c-3bb7-4404-be60-6992b66de9ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:01.905401 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:01.905359 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tls-certs" pod "95fd0e1c-3bb7-4404-be60-6992b66de9ee" (UID: "95fd0e1c-3bb7-4404-be60-6992b66de9ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:02.004683 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.004642 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fd0e1c-3bb7-4404-be60-6992b66de9ee-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:02.004683 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.004679 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fd0e1c-3bb7-4404-be60-6992b66de9ee-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:02.005003 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.004695 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvp4r\" (UniqueName: \"kubernetes.io/projected/95fd0e1c-3bb7-4404-be60-6992b66de9ee-kube-api-access-nvp4r\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:02.086939 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.086905 4898 generic.go:356] "Generic (PLEG): container finished" podID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerID="7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453" exitCode=0 Nov 03 23:02:02.087150 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.086956 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" Nov 03 23:02:02.087150 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.086958 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" event={"ID":"95fd0e1c-3bb7-4404-be60-6992b66de9ee","Type":"ContainerDied","Data":"7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453"} Nov 03 23:02:02.087150 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.087073 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6" event={"ID":"95fd0e1c-3bb7-4404-be60-6992b66de9ee","Type":"ContainerDied","Data":"34f59e874c9181d21d1b4dcd630ae1590c5aaa5ea5f7a5a705201150e24999fe"} Nov 03 23:02:02.087150 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.087098 4898 scope.go:122] "RemoveContainer" containerID="7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453" Nov 03 23:02:02.097854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.097835 4898 scope.go:122] "RemoveContainer" containerID="7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453" Nov 03 23:02:02.098161 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:02:02.098141 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453\": container with ID starting with 7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453 not found: ID does not exist" containerID="7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453" Nov 03 23:02:02.098219 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.098170 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453"} err="failed to get container status \"7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453\": rpc error: code = NotFound desc = could not find container \"7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453\": container with ID starting with 7cb42d229ffa94d016ab330348af4cc9e67a91dab78a53c09c750ee4857de453 not found: ID does not exist" Nov 03 23:02:02.255999 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.255924 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6"] Nov 03 23:02:02.262922 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.262888 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-545b5d/distinct-w1-545b5d-kserve-5957478b5f-7rtv6"] Nov 03 23:02:02.906298 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:02.906262 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" path="/var/lib/kubelet/pods/95fd0e1c-3bb7-4404-be60-6992b66de9ee/volumes" Nov 03 23:02:06.885993 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.885954 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz"] Nov 03 23:02:06.886769 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.886746 4898 cpu_manager.go:414] "containerMap: removing container" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" Nov 03 23:02:06.886902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.886769 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" Nov 03 23:02:06.886902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.886812 4898 cpu_manager.go:414] "containerMap: removing container" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" Nov 03 23:02:06.886902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.886821 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" Nov 03 23:02:06.887055 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.887013 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" Nov 03 23:02:06.887055 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.887029 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fd0e1c-3bb7-4404-be60-6992b66de9ee" containerName="main" Nov 03 23:02:06.887055 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.887041 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" Nov 03 23:02:06.890241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.890218 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:06.993412 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.993379 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0bf80c-serving-cert\"" Nov 03 23:02:06.996483 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:06.996453 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz"] Nov 03 23:02:07.065734 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.065696 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840943c2-edc7-4b62-bb37-c1b5881a743b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.065932 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.065738 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55pb\" (UniqueName: \"kubernetes.io/projected/840943c2-edc7-4b62-bb37-c1b5881a743b-kube-api-access-s55pb\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.066024 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.066006 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-metrics-certs\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.066076 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.066040 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.167503 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.167387 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-metrics-certs\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.167503 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.167443 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.167755 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.167518 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840943c2-edc7-4b62-bb37-c1b5881a743b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.167755 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.167558 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55pb\" (UniqueName: \"kubernetes.io/projected/840943c2-edc7-4b62-bb37-c1b5881a743b-kube-api-access-s55pb\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.171648 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:02:07.168626 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-serving-cert: secret "maas-api-e2e-inf-a-0bf80c-serving-cert" not found Nov 03 23:02:07.171648 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:02:07.168697 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls podName:840943c2-edc7-4b62-bb37-c1b5881a743b nodeName:}" failed. No retries permitted until 2026-11-03 23:02:07.668677134 +0000 UTC m=+2370.881446179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls") pod "maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" (UID: "840943c2-edc7-4b62-bb37-c1b5881a743b") : secret "maas-api-e2e-inf-a-0bf80c-serving-cert" not found Nov 03 23:02:07.171648 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.170340 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840943c2-edc7-4b62-bb37-c1b5881a743b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.183372 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.183338 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-metrics-certs\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.229118 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.229097 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55pb\" (UniqueName: \"kubernetes.io/projected/840943c2-edc7-4b62-bb37-c1b5881a743b-kube-api-access-s55pb\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.673690 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.673645 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.676254 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.676229 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:07.802717 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:07.802681 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:08.180484 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:02:08.180439 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840943c2_edc7_4b62_bb37_c1b5881a743b.slice/crio-0cae46c2f4f4a935162731a8968822dc0a7b3b737bd90eebfd494bae1b36e37d WatchSource:0}: Error finding container 0cae46c2f4f4a935162731a8968822dc0a7b3b737bd90eebfd494bae1b36e37d: Status 404 returned error can't find the container with id 0cae46c2f4f4a935162731a8968822dc0a7b3b737bd90eebfd494bae1b36e37d Nov 03 23:02:08.180928 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:08.180687 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz"] Nov 03 23:02:09.165868 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:09.165820 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" event={"ID":"840943c2-edc7-4b62-bb37-c1b5881a743b","Type":"ContainerStarted","Data":"0cae46c2f4f4a935162731a8968822dc0a7b3b737bd90eebfd494bae1b36e37d"} Nov 03 23:02:10.176389 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:10.176350 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" event={"ID":"840943c2-edc7-4b62-bb37-c1b5881a743b","Type":"ContainerStarted","Data":"384c8f9e98f4ef56945d18b16ebaab65e0174c12be43c88ede638bcb002f3b8f"} Nov 03 23:02:10.176899 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:10.176446 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:10.294677 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:10.294617 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" podStartSLOduration=3.273359314 podStartE2EDuration="4.294598658s" totalImagesPullingTime="1.021239344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:10.293989302 +0000 UTC m=+2373.506758364" watchObservedRunningTime="2026-11-03 23:02:10.294598658 +0000 UTC m=+2373.507367710" Nov 03 23:02:11.810951 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.810903 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs"] Nov 03 23:02:11.811585 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.811567 4898 cpu_manager.go:414] "containerMap: removing container" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" Nov 03 23:02:11.811651 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.811586 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2267cf-8669-41cd-acbd-8f02e561853e" containerName="istio-proxy" Nov 03 23:02:11.814369 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.814350 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.821110 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.821079 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-metrics-certs\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.821214 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.821135 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-openshift-service-ca\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.821265 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.821243 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvszf\" (UniqueName: \"kubernetes.io/projected/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-kube-api-access-gvszf\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.821302 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.821285 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.867710 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.867681 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-656ee6-serving-cert\"" Nov 03 23:02:11.920805 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.920759 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs"] Nov 03 23:02:11.922921 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.922896 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-metrics-certs\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.923042 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.922974 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-openshift-service-ca\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.923112 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.923058 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvszf\" (UniqueName: \"kubernetes.io/projected/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-kube-api-access-gvszf\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.923112 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.923104 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.923271 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:02:11.923248 4898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-serving-cert: secret "maas-api-e2e-api-b-656ee6-serving-cert" not found Nov 03 23:02:11.923375 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:02:11.923358 4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls podName:e6a938fa-5b52-4d3c-bd23-1497ec55fd84 nodeName:}" failed. No retries permitted until 2026-11-03 23:02:12.423336571 +0000 UTC m=+2375.636105600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls") pod "maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" (UID: "e6a938fa-5b52-4d3c-bd23-1497ec55fd84") : secret "maas-api-e2e-api-b-656ee6-serving-cert" not found Nov 03 23:02:11.923753 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.923734 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-openshift-service-ca\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.925617 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.925597 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-metrics-certs\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:11.984869 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:11.984817 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvszf\" (UniqueName: \"kubernetes.io/projected/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-kube-api-access-gvszf\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:12.430016 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:12.429975 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:12.432434 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:12.432405 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls\") pod \"maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:12.727327 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:12.727242 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:13.124956 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:13.124924 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs"] Nov 03 23:02:13.125623 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:02:13.125594 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a938fa_5b52_4d3c_bd23_1497ec55fd84.slice/crio-0f33462b757bebeb223e47ab96880213fe27648748737e97c3788dca3a5fe5d6 WatchSource:0}: Error finding container 0f33462b757bebeb223e47ab96880213fe27648748737e97c3788dca3a5fe5d6: Status 404 returned error can't find the container with id 0f33462b757bebeb223e47ab96880213fe27648748737e97c3788dca3a5fe5d6 Nov 03 23:02:13.213380 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:13.213278 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" event={"ID":"e6a938fa-5b52-4d3c-bd23-1497ec55fd84","Type":"ContainerStarted","Data":"0f33462b757bebeb223e47ab96880213fe27648748737e97c3788dca3a5fe5d6"} Nov 03 23:02:14.222188 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:14.222151 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" event={"ID":"e6a938fa-5b52-4d3c-bd23-1497ec55fd84","Type":"ContainerStarted","Data":"6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253"} Nov 03 23:02:14.222595 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:14.222219 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:14.341800 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:14.341738 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" podStartSLOduration=2.4521209219999998 podStartE2EDuration="3.341723312s" totalImagesPullingTime="889.60239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:14.339749253 +0000 UTC m=+2377.552518325" watchObservedRunningTime="2026-11-03 23:02:14.341723312 +0000 UTC m=+2377.554492362" Nov 03 23:02:16.188280 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:16.188250 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:02:19.837834 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:02:19.837540 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:02:20.234354 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:20.234281 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:02:30.012488 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:30.012446 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq"] Nov 03 23:02:30.013012 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:30.012705 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" containerID="cri-o://97998fdebcc28f21cdece0c060b343a287789c0cc75be0e5c824f203d1ac96af" gracePeriod=30 Nov 03 23:02:35.383167 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.383136 4898 generic.go:356] "Generic (PLEG): container finished" podID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerID="97998fdebcc28f21cdece0c060b343a287789c0cc75be0e5c824f203d1ac96af" exitCode=0 Nov 03 23:02:35.383525 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.383207 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" event={"ID":"fe33eaa6-3be4-46a5-80df-db33aee29a68","Type":"ContainerDied","Data":"97998fdebcc28f21cdece0c060b343a287789c0cc75be0e5c824f203d1ac96af"} Nov 03 23:02:35.383525 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.383256 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" event={"ID":"fe33eaa6-3be4-46a5-80df-db33aee29a68","Type":"ContainerDied","Data":"fc36c8c254c98e8f719ee912ba16469a0c08681e39b1a66bc19e0841eb96539e"} Nov 03 23:02:35.383525 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.383277 4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc36c8c254c98e8f719ee912ba16469a0c08681e39b1a66bc19e0841eb96539e" Nov 03 23:02:35.394203 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.394179 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 23:02:35.527950 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.527918 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-envoy\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528139 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.527990 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe33eaa6-3be4-46a5-80df-db33aee29a68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33eaa6-3be4-46a5-80df-db33aee29a68-istiod-ca-cert\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528139 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528100 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-kube-api-access-n4frc\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-kube-api-access-n4frc\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528267 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528148 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-token\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-token\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528267 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528190 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-credential-socket\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528267 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528257 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-data\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528308 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-podinfo\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528381 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-certs\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528529 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528416 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-socket\") pod \"fe33eaa6-3be4-46a5-80df-db33aee29a68\" (UID: \"fe33eaa6-3be4-46a5-80df-db33aee29a68\") " Nov 03 23:02:35.528581 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528552 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe33eaa6-3be4-46a5-80df-db33aee29a68-istiod-ca-cert" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:35.529381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.528753 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-credential-socket" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:35.529381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.529006 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-socket" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:35.529381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.529045 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33eaa6-3be4-46a5-80df-db33aee29a68-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.529381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.529070 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.529381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.529206 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-certs" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:35.529381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.529326 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-data" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:35.530570 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.530537 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-token" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:35.530676 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.530647 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-envoy" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:35.530999 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.530974 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-podinfo" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:02:35.532117 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.532097 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-kube-api-access-n4frc" pod "fe33eaa6-3be4-46a5-80df-db33aee29a68" (UID: "fe33eaa6-3be4-46a5-80df-db33aee29a68"). InnerVolumeSpecName "kube-api-access-n4frc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:35.629995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.629922 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.629995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.629948 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.629995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.629958 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.629995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.629968 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4frc\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-kube-api-access-n4frc\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.629995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.629977 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.629995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.629986 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:35.629995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:35.629995 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33eaa6-3be4-46a5-80df-db33aee29a68-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:36.390194 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:36.390161 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq" Nov 03 23:02:36.615666 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:36.615635 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq"] Nov 03 23:02:36.618450 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:36.618425 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-b69rq"] Nov 03 23:02:36.908168 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:36.908138 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" path="/var/lib/kubelet/pods/fe33eaa6-3be4-46a5-80df-db33aee29a68/volumes" Nov 03 23:02:37.505656 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:37.505626 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:02:37.511650 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:37.511630 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:02:37.587814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:37.587757 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:02:37.592230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:37.592204 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:02:39.462416 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:39.462373 4898 scope.go:122] "RemoveContainer" containerID="97998fdebcc28f21cdece0c060b343a287789c0cc75be0e5c824f203d1ac96af" Nov 03 23:02:43.117491 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.117417 4898 patch_prober.go:29] interesting pod/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 23:02:43.117938 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.117482 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 23:02:43.433577 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.433542 4898 generic.go:356] "Generic (PLEG): container finished" podID="c9b11632-5041-429c-aabf-23b9123082e2" containerID="b18ee4668f759b81194cac1011c272db830a336d4cd538f79d490185814c8c82" exitCode=0 Nov 03 23:02:43.433773 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.433652 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" event={"ID":"c9b11632-5041-429c-aabf-23b9123082e2","Type":"ContainerDied","Data":"b18ee4668f759b81194cac1011c272db830a336d4cd538f79d490185814c8c82"} Nov 03 23:02:43.434177 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.434160 4898 scope.go:122] "RemoveContainer" containerID="b18ee4668f759b81194cac1011c272db830a336d4cd538f79d490185814c8c82" Nov 03 23:02:43.435847 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.435808 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:43.435986 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.435894 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:43.435986 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:43.435933 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:44.448812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:44.448403 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" event={"ID":"c9b11632-5041-429c-aabf-23b9123082e2","Type":"ContainerStarted","Data":"de63c105240dea39795b11dfd8d626feb5f9b8d76951a9fd672f699a388656e6"} Nov 03 23:02:45.182201 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:45.182161 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:02:45.182451 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:45.182425 4898 patch_prober.go:29] interesting pod/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 23:02:45.182579 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:45.182556 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 23:02:46.181367 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:46.181328 4898 patch_prober.go:29] interesting pod/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 23:02:46.181809 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:46.181398 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 23:02:47.181365 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:47.181323 4898 patch_prober.go:29] interesting pod/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 23:02:47.181587 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:47.181394 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 23:02:48.168810 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.168547 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv"] Nov 03 23:02:48.169628 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.169122 4898 cpu_manager.go:414] "containerMap: removing container" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" Nov 03 23:02:48.169628 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.169136 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" Nov 03 23:02:48.169628 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.169259 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe33eaa6-3be4-46a5-80df-db33aee29a68" containerName="istio-proxy" Nov 03 23:02:48.172617 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.172592 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.182154 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.182125 4898 patch_prober.go:29] interesting pod/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 23:02:48.182461 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.182175 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 23:02:48.222092 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.222053 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tls-certs\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.222243 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.222121 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-home\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.222919 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.222886 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-model-cache\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.223114 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.223095 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tmp-dir\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.223249 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.223238 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdc8\" (UniqueName: \"kubernetes.io/projected/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-kube-api-access-jjdc8\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.223425 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.223412 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-dshm\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.230059 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.229854 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv"] Nov 03 23:02:48.275373 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.275346 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2cea18\"/\"default-dockercfg-g9w6c\"" Nov 03 23:02:48.275859 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.275843 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2cea18\"/\"test-model-2cea18-kserve-self-signed-certs\"" Nov 03 23:02:48.276175 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.276160 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2cea18\"/\"openshift-service-ca.crt\"" Nov 03 23:02:48.276778 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.276757 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2cea18\"/\"kube-root-ca.crt\"" Nov 03 23:02:48.324657 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.324617 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-model-cache\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.324872 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.324678 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tmp-dir\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.324872 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.324726 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdc8\" (UniqueName: \"kubernetes.io/projected/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-kube-api-access-jjdc8\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.324872 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.324756 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-dshm\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.325047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.324895 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tls-certs\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.325047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.324944 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-home\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.325175 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.325119 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-model-cache\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.325175 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.325139 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tmp-dir\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.325300 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.325276 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-home\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.327044 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.327022 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-dshm\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.327490 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.327471 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tls-certs\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.388829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.386724 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdc8\" (UniqueName: \"kubernetes.io/projected/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-kube-api-access-jjdc8\") pod \"test-model-2cea18-kserve-c8bc5fc8c-tbfzv\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.486270 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.486098 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:48.984685 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:48.984652 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv"] Nov 03 23:02:48.985073 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:02:48.985045 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d33dd8f_92b5_4625_8ff0_3595e5d3bd11.slice/crio-1ca4e3a615fc9d0e1456e74b92cbbe357b89fe93f3489e0079408f5683a5e5ae WatchSource:0}: Error finding container 1ca4e3a615fc9d0e1456e74b92cbbe357b89fe93f3489e0079408f5683a5e5ae: Status 404 returned error can't find the container with id 1ca4e3a615fc9d0e1456e74b92cbbe357b89fe93f3489e0079408f5683a5e5ae Nov 03 23:02:49.180566 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:49.180525 4898 patch_prober.go:29] interesting pod/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 23:02:49.180749 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:49.180601 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 23:02:49.497485 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:49.497445 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" event={"ID":"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11","Type":"ContainerStarted","Data":"3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da"} Nov 03 23:02:49.497945 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:49.497498 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" event={"ID":"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11","Type":"ContainerStarted","Data":"1ca4e3a615fc9d0e1456e74b92cbbe357b89fe93f3489e0079408f5683a5e5ae"} Nov 03 23:02:49.617129 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:49.617070 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" podStartSLOduration=1.6170559660000001 podStartE2EDuration="1.617055966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:49.614430692 +0000 UTC m=+2412.827199764" watchObservedRunningTime="2026-11-03 23:02:49.617055966 +0000 UTC m=+2412.829825032" Nov 03 23:02:49.836891 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:02:49.836843 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:02:50.185363 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:50.185282 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:02:50.185738 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:50.185724 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:02:50.186591 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:50.186571 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:02:50.486668 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:50.486592 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:57.162857 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:57.162805 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b"] Nov 03 23:02:58.119418 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.119383 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs"] Nov 03 23:02:58.123695 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.123667 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.177817 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.177217 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-545b5d-openshift-default-dockercfg-t5wb5\"" Nov 03 23:02:58.225268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225235 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225272 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225472 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225344 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225516 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225484 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225585 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225571 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225628 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225614 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225663 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225640 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225663 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225657 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.225854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.225832 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqq5\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-kube-api-access-gvqq5\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.232308 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.232282 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs"] Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328540 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328668 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328701 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328735 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328767 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328861 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqq5\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-kube-api-access-gvqq5\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328904 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328941 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.329268 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.328966 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.330814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.330189 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.331812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.330969 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.331812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.331236 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.331812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.331476 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.340295 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.340238 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.340403 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.340310 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.342483 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.342460 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.402576 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.402504 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.402928 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.402870 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqq5\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-kube-api-access-gvqq5\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.440716 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.440686 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:02:58.487074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.487037 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:58.513835 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.511679 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:58.590144 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.590120 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:02:58.919021 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.918990 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs"] Nov 03 23:02:58.919976 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:02:58.919948 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4f1c63_7c5f_4df2_a48e_b1c9204d313c.slice/crio-0238a89f729a9b5b7d57b858eb9960ed8e1c0c99eb75e9a0baa8bd391218d651 WatchSource:0}: Error finding container 0238a89f729a9b5b7d57b858eb9960ed8e1c0c99eb75e9a0baa8bd391218d651: Status 404 returned error can't find the container with id 0238a89f729a9b5b7d57b858eb9960ed8e1c0c99eb75e9a0baa8bd391218d651 Nov 03 23:02:58.922335 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.922298 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:58.922431 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.922375 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:58.922431 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:58.922419 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:59.577230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:59.577186 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" event={"ID":"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c","Type":"ContainerStarted","Data":"19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb"} Nov 03 23:02:59.577735 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:59.577250 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" event={"ID":"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c","Type":"ContainerStarted","Data":"0238a89f729a9b5b7d57b858eb9960ed8e1c0c99eb75e9a0baa8bd391218d651"} Nov 03 23:02:59.699524 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:02:59.699468 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podStartSLOduration=1.6994533029999999 podStartE2EDuration="1.699453303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:59.698162119 +0000 UTC m=+2422.910931171" watchObservedRunningTime="2026-11-03 23:02:59.699453303 +0000 UTC m=+2422.912222421" Nov 03 23:03:00.458328 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:00.446250 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:03:00.460987 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:00.460944 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 23:03:00.461141 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:00.461018 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 23:03:01.453813 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:01.446843 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 23:03:01.453813 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:01.446905 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 23:03:02.447812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:02.441200 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 23:03:02.447812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:02.441267 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 23:03:03.456812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:03.447387 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 23:03:03.456812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:03.447462 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 23:03:03.986164 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:03.986117 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495"] Nov 03 23:03:04.458979 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:04.458926 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 23:03:04.459407 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:04.458990 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 23:03:04.882803 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:04.882735 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb"] Nov 03 23:03:04.884301 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:04.884262 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" podUID="82a88795-e98d-4780-b672-3a6dee25b4d3" containerName="maas-api" containerID="cri-o://4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893" gracePeriod=30 Nov 03 23:03:05.257882 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.257591 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 23:03:05.341044 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.341003 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a88795-e98d-4780-b672-3a6dee25b4d3-kube-api-access-tpjm5\" (UniqueName: \"kubernetes.io/projected/82a88795-e98d-4780-b672-3a6dee25b4d3-kube-api-access-tpjm5\") pod \"82a88795-e98d-4780-b672-3a6dee25b4d3\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " Nov 03 23:03:05.341231 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.341093 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a88795-e98d-4780-b672-3a6dee25b4d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a88795-e98d-4780-b672-3a6dee25b4d3-openshift-service-ca\") pod \"82a88795-e98d-4780-b672-3a6dee25b4d3\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " Nov 03 23:03:05.341231 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.341182 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-metrics-certs\") pod \"82a88795-e98d-4780-b672-3a6dee25b4d3\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " Nov 03 23:03:05.341354 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.341311 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls\") pod \"82a88795-e98d-4780-b672-3a6dee25b4d3\" (UID: \"82a88795-e98d-4780-b672-3a6dee25b4d3\") " Nov 03 23:03:05.341492 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.341463 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a88795-e98d-4780-b672-3a6dee25b4d3-openshift-service-ca" pod "82a88795-e98d-4780-b672-3a6dee25b4d3" (UID: "82a88795-e98d-4780-b672-3a6dee25b4d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:05.341842 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.341782 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a88795-e98d-4780-b672-3a6dee25b4d3-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:05.343385 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.343349 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-metrics-certs" pod "82a88795-e98d-4780-b672-3a6dee25b4d3" (UID: "82a88795-e98d-4780-b672-3a6dee25b4d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:05.343500 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.343426 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls" pod "82a88795-e98d-4780-b672-3a6dee25b4d3" (UID: "82a88795-e98d-4780-b672-3a6dee25b4d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:05.343980 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.343957 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a88795-e98d-4780-b672-3a6dee25b4d3-kube-api-access-tpjm5" pod "82a88795-e98d-4780-b672-3a6dee25b4d3" (UID: "82a88795-e98d-4780-b672-3a6dee25b4d3"). InnerVolumeSpecName "kube-api-access-tpjm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:05.441596 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.441555 4898 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 23:03:05.441825 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.441608 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 23:03:05.442508 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.442484 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjm5\" (UniqueName: \"kubernetes.io/projected/82a88795-e98d-4780-b672-3a6dee25b4d3-kube-api-access-tpjm5\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:05.442508 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.442507 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:05.442680 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.442516 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a88795-e98d-4780-b672-3a6dee25b4d3-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:05.662825 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.662180 4898 generic.go:356] "Generic (PLEG): container finished" podID="82a88795-e98d-4780-b672-3a6dee25b4d3" containerID="4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893" exitCode=0 Nov 03 23:03:05.662825 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.662305 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" event={"ID":"82a88795-e98d-4780-b672-3a6dee25b4d3","Type":"ContainerDied","Data":"4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893"} Nov 03 23:03:05.662825 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.662360 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" event={"ID":"82a88795-e98d-4780-b672-3a6dee25b4d3","Type":"ContainerDied","Data":"efb242275650fdfdfc479330cdcd667a0e5ee8d2680bd881ba10b85dd04b8c1f"} Nov 03 23:03:05.662825 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.662390 4898 scope.go:122] "RemoveContainer" containerID="4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893" Nov 03 23:03:05.662825 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.662591 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb" Nov 03 23:03:05.679802 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.679766 4898 scope.go:122] "RemoveContainer" containerID="4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893" Nov 03 23:03:05.680199 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:05.680168 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893\": container with ID starting with 4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893 not found: ID does not exist" containerID="4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893" Nov 03 23:03:05.680319 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.680212 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893"} err="failed to get container status \"4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893\": rpc error: code = NotFound desc = could not find container \"4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893\": container with ID starting with 4eceee225792dcbfb3208e2feabd6d08d42ad6b6116d07797ded6de46c128893 not found: ID does not exist" Nov 03 23:03:05.844303 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.842672 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb"] Nov 03 23:03:05.849376 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:05.849344 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-545b5d-84d9b77c95-kf5tb"] Nov 03 23:03:06.058283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.058188 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk"] Nov 03 23:03:06.058909 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.058435 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" podUID="670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" containerName="maas-api" containerID="cri-o://fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c" gracePeriod=30 Nov 03 23:03:06.481813 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.480316 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 23:03:06.554040 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.554007 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:03:06.554381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.554360 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:03:06.555382 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.555360 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:03:06.573172 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.573145 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls\") pod \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " Nov 03 23:03:06.573316 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.573182 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-kube-api-access-sftk4\" (UniqueName: \"kubernetes.io/projected/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-kube-api-access-sftk4\") pod \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " Nov 03 23:03:06.573316 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.573213 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-openshift-service-ca\") pod \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " Nov 03 23:03:06.573316 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.573296 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-metrics-certs\") pod \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\" (UID: \"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa\") " Nov 03 23:03:06.574104 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.574065 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-openshift-service-ca" pod "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" (UID: "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:06.578992 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.578962 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-kube-api-access-sftk4" pod "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" (UID: "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa"). InnerVolumeSpecName "kube-api-access-sftk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:06.580575 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.580542 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls" pod "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" (UID: "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:06.580860 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.580831 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-metrics-certs" pod "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" (UID: "670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:06.670766 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.670729 4898 generic.go:356] "Generic (PLEG): container finished" podID="670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" containerID="fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c" exitCode=0 Nov 03 23:03:06.671234 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.670782 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" Nov 03 23:03:06.671234 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.670851 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" event={"ID":"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa","Type":"ContainerDied","Data":"fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c"} Nov 03 23:03:06.671234 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.670914 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk" event={"ID":"670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa","Type":"ContainerDied","Data":"88746b4f636f267838a84998fc6ada0f530f206b92fa96ae1ee0408e4cbd810a"} Nov 03 23:03:06.671234 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.670952 4898 scope.go:122] "RemoveContainer" containerID="fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c" Nov 03 23:03:06.674674 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.674646 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:06.674808 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.674677 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sftk4\" (UniqueName: \"kubernetes.io/projected/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-kube-api-access-sftk4\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:06.674808 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.674695 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:06.674808 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.674708 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:06.682542 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.682522 4898 scope.go:122] "RemoveContainer" containerID="fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c" Nov 03 23:03:06.682909 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:06.682881 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c\": container with ID starting with fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c not found: ID does not exist" containerID="fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c" Nov 03 23:03:06.682984 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.682922 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c"} err="failed to get container status \"fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c\": rpc error: code = NotFound desc = could not find container \"fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c\": container with ID starting with fe469a7d141e97309f5ee35a8de71173024f02c14ce817400d867ce08cec432c not found: ID does not exist" Nov 03 23:03:06.839190 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.839147 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk"] Nov 03 23:03:06.847116 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.847082 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ee74cb-5c5598dd95-g9dtk"] Nov 03 23:03:06.915137 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.910741 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" path="/var/lib/kubelet/pods/670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa/volumes" Nov 03 23:03:06.915137 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:06.911584 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a88795-e98d-4780-b672-3a6dee25b4d3" path="/var/lib/kubelet/pods/82a88795-e98d-4780-b672-3a6dee25b4d3/volumes" Nov 03 23:03:08.987396 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:08.987334 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" podUID="ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" containerName="payload-pre-processing" containerID="cri-o://cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d" gracePeriod=25 Nov 03 23:03:09.288178 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.288153 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 23:03:09.444223 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.444192 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-plugins-config-volume\") pod \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " Nov 03 23:03:09.444424 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.444284 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-kube-api-access-77czh\" (UniqueName: \"kubernetes.io/projected/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-kube-api-access-77czh\") pod \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\" (UID: \"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da\") " Nov 03 23:03:09.445326 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.445291 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-plugins-config-volume" pod "ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" (UID: "ec5068bb-6e95-43f3-a4ef-e7f27fdca4da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:09.446821 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.446758 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-kube-api-access-77czh" pod "ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" (UID: "ec5068bb-6e95-43f3-a4ef-e7f27fdca4da"). InnerVolumeSpecName "kube-api-access-77czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:09.546307 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.546219 4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-plugins-config-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:09.546307 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.546253 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77czh\" (UniqueName: \"kubernetes.io/projected/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da-kube-api-access-77czh\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:09.718227 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.718193 4898 generic.go:356] "Generic (PLEG): container finished" podID="ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" containerID="cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d" exitCode=0 Nov 03 23:03:09.718399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.718236 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" event={"ID":"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da","Type":"ContainerDied","Data":"cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d"} Nov 03 23:03:09.718399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.718241 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" Nov 03 23:03:09.718399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.718278 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495" event={"ID":"ec5068bb-6e95-43f3-a4ef-e7f27fdca4da","Type":"ContainerDied","Data":"a067f12abb269f5d1fc43b64159be6eb6e6f69b3b6b65e6c8620953f36e7d7cb"} Nov 03 23:03:09.718399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.718298 4898 scope.go:122] "RemoveContainer" containerID="cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d" Nov 03 23:03:09.728609 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.728583 4898 scope.go:122] "RemoveContainer" containerID="cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d" Nov 03 23:03:09.728912 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:09.728892 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d\": container with ID starting with cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d not found: ID does not exist" containerID="cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d" Nov 03 23:03:09.728973 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.728921 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d"} err="failed to get container status \"cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d\": rpc error: code = NotFound desc = could not find container \"cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d\": container with ID starting with cecfd821b07f5be32a5ab156dc93ae9f606d35c9d62b8fdb0fcc0fd3ad19965d not found: ID does not exist" Nov 03 23:03:09.885011 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.884969 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495"] Nov 03 23:03:09.888427 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:09.888400 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18c03a-d7d874c9-bn495"] Nov 03 23:03:10.906723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:10.906692 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" path="/var/lib/kubelet/pods/ec5068bb-6e95-43f3-a4ef-e7f27fdca4da/volumes" Nov 03 23:03:12.187216 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.187168 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" containerID="cri-o://c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa" gracePeriod=45 Nov 03 23:03:12.465988 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.465961 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 23:03:12.482145 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482114 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-tmp-dir\") pod \"d6f23050-48f9-4b70-b3e5-0794b749e533\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " Nov 03 23:03:12.482287 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482179 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6f23050-48f9-4b70-b3e5-0794b749e533-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f23050-48f9-4b70-b3e5-0794b749e533-tls-certs\") pod \"d6f23050-48f9-4b70-b3e5-0794b749e533\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " Nov 03 23:03:12.482287 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482223 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-model-cache\") pod \"d6f23050-48f9-4b70-b3e5-0794b749e533\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " Nov 03 23:03:12.482396 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482346 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-home\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-home\") pod \"d6f23050-48f9-4b70-b3e5-0794b749e533\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " Nov 03 23:03:12.482396 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482359 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-tmp-dir" pod "d6f23050-48f9-4b70-b3e5-0794b749e533" (UID: "d6f23050-48f9-4b70-b3e5-0794b749e533"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.482553 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482499 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-model-cache" pod "d6f23050-48f9-4b70-b3e5-0794b749e533" (UID: "d6f23050-48f9-4b70-b3e5-0794b749e533"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.482641 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482573 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-home" pod "d6f23050-48f9-4b70-b3e5-0794b749e533" (UID: "d6f23050-48f9-4b70-b3e5-0794b749e533"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.482821 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482802 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.482902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482827 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.482902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.482841 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.484429 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.484409 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f23050-48f9-4b70-b3e5-0794b749e533-tls-certs" pod "d6f23050-48f9-4b70-b3e5-0794b749e533" (UID: "d6f23050-48f9-4b70-b3e5-0794b749e533"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:12.583357 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.583321 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-dshm\") pod \"d6f23050-48f9-4b70-b3e5-0794b749e533\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " Nov 03 23:03:12.583541 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.583394 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f23050-48f9-4b70-b3e5-0794b749e533-kube-api-access-kbh2f\" (UniqueName: \"kubernetes.io/projected/d6f23050-48f9-4b70-b3e5-0794b749e533-kube-api-access-kbh2f\") pod \"d6f23050-48f9-4b70-b3e5-0794b749e533\" (UID: \"d6f23050-48f9-4b70-b3e5-0794b749e533\") " Nov 03 23:03:12.584475 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.584434 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f23050-48f9-4b70-b3e5-0794b749e533-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.586499 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.586469 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-dshm" pod "d6f23050-48f9-4b70-b3e5-0794b749e533" (UID: "d6f23050-48f9-4b70-b3e5-0794b749e533"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.586914 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.586883 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f23050-48f9-4b70-b3e5-0794b749e533-kube-api-access-kbh2f" pod "d6f23050-48f9-4b70-b3e5-0794b749e533" (UID: "d6f23050-48f9-4b70-b3e5-0794b749e533"). InnerVolumeSpecName "kube-api-access-kbh2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:12.685224 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.685193 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f23050-48f9-4b70-b3e5-0794b749e533-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.685224 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.685221 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbh2f\" (UniqueName: \"kubernetes.io/projected/d6f23050-48f9-4b70-b3e5-0794b749e533-kube-api-access-kbh2f\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.755864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.748094 4898 generic.go:356] "Generic (PLEG): container finished" podID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerID="c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa" exitCode=0 Nov 03 23:03:12.755864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.748165 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" event={"ID":"d6f23050-48f9-4b70-b3e5-0794b749e533","Type":"ContainerDied","Data":"c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa"} Nov 03 23:03:12.755864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.748204 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" event={"ID":"d6f23050-48f9-4b70-b3e5-0794b749e533","Type":"ContainerDied","Data":"6a370042e34da8698fc7271342bc6991ed8f14e6a2533c413b485e776040a7e6"} Nov 03 23:03:12.755864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.748228 4898 scope.go:122] "RemoveContainer" containerID="c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa" Nov 03 23:03:12.755864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.748353 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b" Nov 03 23:03:12.793157 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.793126 4898 scope.go:122] "RemoveContainer" containerID="c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa" Nov 03 23:03:12.793551 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:12.793526 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa\": container with ID starting with c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa not found: ID does not exist" containerID="c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa" Nov 03 23:03:12.793685 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.793564 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa"} err="failed to get container status \"c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa\": rpc error: code = NotFound desc = could not find container \"c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa\": container with ID starting with c7b04ba12c8b379acee7866a0123aaa0de980692b82e9282828a4a684302bbaa not found: ID does not exist" Nov 03 23:03:12.930808 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.929558 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b"] Nov 03 23:03:12.936805 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:12.934332 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/premium-w4-18c03a-kserve-7787df7cf6-wkr4b"] Nov 03 23:03:14.906380 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:14.906345 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" path="/var/lib/kubelet/pods/d6f23050-48f9-4b70-b3e5-0794b749e533/volumes" Nov 03 23:03:15.914558 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:15.914522 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq"] Nov 03 23:03:15.915091 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:15.914724 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" containerID="cri-o://e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242" gracePeriod=30 Nov 03 23:03:19.834633 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:19.834592 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:03:21.262295 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.262270 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 23:03:21.404916 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.404878 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-envoy\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405115 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.404940 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-credential-socket\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405115 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405000 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-token\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-token\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405276 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405238 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-credential-socket" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.405388 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405035 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-podinfo\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405532 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405507 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-kube-api-access-l6sgq\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-kube-api-access-l6sgq\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405699 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405591 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-data\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405849 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405697 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-socket\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405849 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405735 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d4eff3-033c-4bcf-9fcd-051294b437b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4eff3-033c-4bcf-9fcd-051294b437b3-istiod-ca-cert\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.405849 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405828 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-certs\") pod \"65d4eff3-033c-4bcf-9fcd-051294b437b3\" (UID: \"65d4eff3-033c-4bcf-9fcd-051294b437b3\") " Nov 03 23:03:21.406163 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.405937 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-data" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.406163 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.406133 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-socket" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.406371 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.406350 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.406371 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.406378 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.406371 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.406394 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.406678 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.406418 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d4eff3-033c-4bcf-9fcd-051294b437b3-istiod-ca-cert" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:21.406678 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.406642 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-certs" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.408048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.407996 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-envoy" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.408362 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.408326 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-kube-api-access-l6sgq" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "kube-api-access-l6sgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:21.408902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.408879 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-token" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:21.409583 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.409557 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-podinfo" pod "65d4eff3-033c-4bcf-9fcd-051294b437b3" (UID: "65d4eff3-033c-4bcf-9fcd-051294b437b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:21.508187 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.508097 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4eff3-033c-4bcf-9fcd-051294b437b3-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.508187 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.508134 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.508187 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.508150 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.508187 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.508166 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.508187 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.508180 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4eff3-033c-4bcf-9fcd-051294b437b3-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.508463 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.508194 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6sgq\" (UniqueName: \"kubernetes.io/projected/65d4eff3-033c-4bcf-9fcd-051294b437b3-kube-api-access-l6sgq\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.811871 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.811750 4898 generic.go:356] "Generic (PLEG): container finished" podID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerID="e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242" exitCode=0 Nov 03 23:03:21.811871 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.811834 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" Nov 03 23:03:21.811871 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.811843 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" event={"ID":"65d4eff3-033c-4bcf-9fcd-051294b437b3","Type":"ContainerDied","Data":"e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242"} Nov 03 23:03:21.812168 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.811912 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq" event={"ID":"65d4eff3-033c-4bcf-9fcd-051294b437b3","Type":"ContainerDied","Data":"889e5b12df51d1d7a622d0e5b2757f17bf38d689a98125ab3bfbc77c31975012"} Nov 03 23:03:21.812168 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.811936 4898 scope.go:122] "RemoveContainer" containerID="e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242" Nov 03 23:03:21.822255 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.822237 4898 scope.go:122] "RemoveContainer" containerID="689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3" Nov 03 23:03:21.831819 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.831777 4898 scope.go:122] "RemoveContainer" containerID="e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242" Nov 03 23:03:21.832111 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:21.832087 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242\": container with ID starting with e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242 not found: ID does not exist" containerID="e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242" Nov 03 23:03:21.832167 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.832119 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242"} err="failed to get container status \"e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242\": rpc error: code = NotFound desc = could not find container \"e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242\": container with ID starting with e5c6a1c049f8411c2d9f5278270c39ea5f6dcdafc6c79ad2bb76c8d7e103d242 not found: ID does not exist" Nov 03 23:03:21.832167 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.832135 4898 scope.go:122] "RemoveContainer" containerID="689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3" Nov 03 23:03:21.832373 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:21.832356 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3\": container with ID starting with 689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3 not found: ID does not exist" containerID="689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3" Nov 03 23:03:21.832418 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:21.832378 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3"} err="failed to get container status \"689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3\": rpc error: code = NotFound desc = could not find container \"689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3\": container with ID starting with 689e3b13737a4543f30faf5a0392011c0ed9d515261d0ab7f40d6563dc8031e3 not found: ID does not exist" Nov 03 23:03:22.008758 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:22.008721 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq"] Nov 03 23:03:22.048182 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:22.048141 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-b7xdq"] Nov 03 23:03:22.906622 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:22.906585 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" path="/var/lib/kubelet/pods/65d4eff3-033c-4bcf-9fcd-051294b437b3/volumes" Nov 03 23:03:24.815745 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:24.815710 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs"] Nov 03 23:03:24.816230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:24.816009 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" containerID="cri-o://19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb" gracePeriod=30 Nov 03 23:03:30.097622 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.097593 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:03:30.218763 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.218731 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-kube-api-access-gvqq5\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-kube-api-access-gvqq5\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.218958 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.218823 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-certs\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.218958 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.218866 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-credential-socket\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.218958 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.218916 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-socket\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.218958 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.218953 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-data\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.219186 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219016 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-token\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.219186 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219064 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-podinfo\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.219186 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219171 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-envoy\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.219340 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219217 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istiod-ca-cert\") pod \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\" (UID: \"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c\") " Nov 03 23:03:30.219573 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219510 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-socket" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.219696 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219609 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-credential-socket" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.220115 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219862 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-certs" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.220115 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.219955 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istiod-ca-cert" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:30.220115 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.220079 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-data" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.220560 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.220504 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.220560 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.220527 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.222101 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.222065 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-kube-api-access-gvqq5" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "kube-api-access-gvqq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:30.222101 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.222092 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-token" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:30.222243 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.222124 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-podinfo" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:30.224423 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.224395 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-envoy" pod "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" (UID: "6f4f1c63-7c5f-4df2-a48e-b1c9204d313c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.322074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.321988 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.322074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.322024 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.322074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.322039 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvqq5\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-kube-api-access-gvqq5\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.322074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.322054 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.322074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.322066 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.322384 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.322079 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.322384 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.322108 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.879040 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.879006 4898 generic.go:356] "Generic (PLEG): container finished" podID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerID="19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb" exitCode=0 Nov 03 23:03:30.879218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.879059 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" event={"ID":"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c","Type":"ContainerDied","Data":"19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb"} Nov 03 23:03:30.879218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.879078 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" Nov 03 23:03:30.879218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.879125 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs" event={"ID":"6f4f1c63-7c5f-4df2-a48e-b1c9204d313c","Type":"ContainerDied","Data":"0238a89f729a9b5b7d57b858eb9960ed8e1c0c99eb75e9a0baa8bd391218d651"} Nov 03 23:03:30.879218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.879163 4898 scope.go:122] "RemoveContainer" containerID="19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb" Nov 03 23:03:30.890895 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.890877 4898 scope.go:122] "RemoveContainer" containerID="19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb" Nov 03 23:03:30.891171 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:30.891143 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb\": container with ID starting with 19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb not found: ID does not exist" containerID="19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb" Nov 03 23:03:30.891226 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.891183 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb"} err="failed to get container status \"19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb\": rpc error: code = NotFound desc = could not find container \"19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb\": container with ID starting with 19304e65f3deb795975022bbcadaa42186cf04f61fe986befdfaff148f015ecb not found: ID does not exist" Nov 03 23:03:30.907454 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:30.907428 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn"] Nov 03 23:03:31.016013 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:31.015980 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz"] Nov 03 23:03:31.050120 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:31.049986 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs"] Nov 03 23:03:31.052773 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:31.052749 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-t6mjs"] Nov 03 23:03:32.907609 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:32.907570 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" path="/var/lib/kubelet/pods/6f4f1c63-7c5f-4df2-a48e-b1c9204d313c/volumes" Nov 03 23:03:35.908738 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:35.908679 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" podUID="06e66f7f-49d3-4e1d-9557-b8c57159c381" containerName="payload-processing" containerID="cri-o://0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0" gracePeriod=25 Nov 03 23:03:36.017298 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.017241 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" podUID="0404aee4-fa58-4d0d-9eac-63ac403503d5" containerName="payload-pre-processing" containerID="cri-o://96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928" gracePeriod=25 Nov 03 23:03:36.193566 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.193540 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 23:03:36.301722 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.301695 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e66f7f-49d3-4e1d-9557-b8c57159c381-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e66f7f-49d3-4e1d-9557-b8c57159c381-plugins-config-volume\") pod \"06e66f7f-49d3-4e1d-9557-b8c57159c381\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " Nov 03 23:03:36.301854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.301769 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e66f7f-49d3-4e1d-9557-b8c57159c381-kube-api-access-tqhnx\" (UniqueName: \"kubernetes.io/projected/06e66f7f-49d3-4e1d-9557-b8c57159c381-kube-api-access-tqhnx\") pod \"06e66f7f-49d3-4e1d-9557-b8c57159c381\" (UID: \"06e66f7f-49d3-4e1d-9557-b8c57159c381\") " Nov 03 23:03:36.302048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.302028 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e66f7f-49d3-4e1d-9557-b8c57159c381-plugins-config-volume" pod "06e66f7f-49d3-4e1d-9557-b8c57159c381" (UID: "06e66f7f-49d3-4e1d-9557-b8c57159c381"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:36.302181 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.302167 4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e66f7f-49d3-4e1d-9557-b8c57159c381-plugins-config-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:36.303604 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.303588 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 23:03:36.303734 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.303716 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e66f7f-49d3-4e1d-9557-b8c57159c381-kube-api-access-tqhnx" pod "06e66f7f-49d3-4e1d-9557-b8c57159c381" (UID: "06e66f7f-49d3-4e1d-9557-b8c57159c381"). InnerVolumeSpecName "kube-api-access-tqhnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:36.402983 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.402950 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0404aee4-fa58-4d0d-9eac-63ac403503d5-kube-api-access-vw4rl\" (UniqueName: \"kubernetes.io/projected/0404aee4-fa58-4d0d-9eac-63ac403503d5-kube-api-access-vw4rl\") pod \"0404aee4-fa58-4d0d-9eac-63ac403503d5\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " Nov 03 23:03:36.403132 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.403087 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0404aee4-fa58-4d0d-9eac-63ac403503d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0404aee4-fa58-4d0d-9eac-63ac403503d5-plugins-config-volume\") pod \"0404aee4-fa58-4d0d-9eac-63ac403503d5\" (UID: \"0404aee4-fa58-4d0d-9eac-63ac403503d5\") " Nov 03 23:03:36.403509 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.403487 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhnx\" (UniqueName: \"kubernetes.io/projected/06e66f7f-49d3-4e1d-9557-b8c57159c381-kube-api-access-tqhnx\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:36.403555 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.403491 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0404aee4-fa58-4d0d-9eac-63ac403503d5-plugins-config-volume" pod "0404aee4-fa58-4d0d-9eac-63ac403503d5" (UID: "0404aee4-fa58-4d0d-9eac-63ac403503d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:36.405326 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.405297 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0404aee4-fa58-4d0d-9eac-63ac403503d5-kube-api-access-vw4rl" pod "0404aee4-fa58-4d0d-9eac-63ac403503d5" (UID: "0404aee4-fa58-4d0d-9eac-63ac403503d5"). InnerVolumeSpecName "kube-api-access-vw4rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:36.505267 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.505180 4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0404aee4-fa58-4d0d-9eac-63ac403503d5-plugins-config-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:36.505267 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.505210 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw4rl\" (UniqueName: \"kubernetes.io/projected/0404aee4-fa58-4d0d-9eac-63ac403503d5-kube-api-access-vw4rl\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:36.927885 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.927847 4898 generic.go:356] "Generic (PLEG): container finished" podID="06e66f7f-49d3-4e1d-9557-b8c57159c381" containerID="0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0" exitCode=0 Nov 03 23:03:36.928291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.927898 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" Nov 03 23:03:36.928291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.927918 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" event={"ID":"06e66f7f-49d3-4e1d-9557-b8c57159c381","Type":"ContainerDied","Data":"0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0"} Nov 03 23:03:36.928291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.927976 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn" event={"ID":"06e66f7f-49d3-4e1d-9557-b8c57159c381","Type":"ContainerDied","Data":"4ecc0a13522970682b4f60afa24502807bf66bce5949858ec7e03350175b8743"} Nov 03 23:03:36.928291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.928001 4898 scope.go:122] "RemoveContainer" containerID="0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0" Nov 03 23:03:36.929708 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.929687 4898 generic.go:356] "Generic (PLEG): container finished" podID="0404aee4-fa58-4d0d-9eac-63ac403503d5" containerID="96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928" exitCode=0 Nov 03 23:03:36.929829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.929718 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" event={"ID":"0404aee4-fa58-4d0d-9eac-63ac403503d5","Type":"ContainerDied","Data":"96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928"} Nov 03 23:03:36.929829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.929756 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" event={"ID":"0404aee4-fa58-4d0d-9eac-63ac403503d5","Type":"ContainerDied","Data":"e4f52e9f637decc00b8fa48539f2db333759ff547006c6818ef895c715e4efeb"} Nov 03 23:03:36.929829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.929766 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz" Nov 03 23:03:36.939474 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.939452 4898 scope.go:122] "RemoveContainer" containerID="0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0" Nov 03 23:03:36.939769 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:36.939749 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0\": container with ID starting with 0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0 not found: ID does not exist" containerID="0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0" Nov 03 23:03:36.939885 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.939774 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0"} err="failed to get container status \"0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0\": rpc error: code = NotFound desc = could not find container \"0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0\": container with ID starting with 0571a70348695fcdc3c317392e9e29ad55ed64c2983dcc0ef23aa9f3665a45d0 not found: ID does not exist" Nov 03 23:03:36.939885 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.939806 4898 scope.go:122] "RemoveContainer" containerID="96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928" Nov 03 23:03:36.949071 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.949046 4898 scope.go:122] "RemoveContainer" containerID="96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928" Nov 03 23:03:36.949387 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:36.949362 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928\": container with ID starting with 96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928 not found: ID does not exist" containerID="96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928" Nov 03 23:03:36.949478 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:36.949396 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928"} err="failed to get container status \"96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928\": rpc error: code = NotFound desc = could not find container \"96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928\": container with ID starting with 96069fbd69c1f9b4a45673170156bc6b280d5fbe3ae84890ce5f2de08b039928 not found: ID does not exist" Nov 03 23:03:37.093999 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:37.093965 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn"] Nov 03 23:03:37.098235 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:37.098210 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-31ad7b-74bb7bcdf8-fdhjn"] Nov 03 23:03:37.208995 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:37.208898 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz"] Nov 03 23:03:37.212138 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:37.212109 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-31ad7b-5c4dcf55-2fmdz"] Nov 03 23:03:38.906744 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:38.906710 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0404aee4-fa58-4d0d-9eac-63ac403503d5" path="/var/lib/kubelet/pods/0404aee4-fa58-4d0d-9eac-63ac403503d5/volumes" Nov 03 23:03:38.907284 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:03:38.907261 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e66f7f-49d3-4e1d-9557-b8c57159c381" path="/var/lib/kubelet/pods/06e66f7f-49d3-4e1d-9557-b8c57159c381/volumes" Nov 03 23:03:49.834710 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:03:49.834670 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:04:16.958734 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:16.958698 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz"] Nov 03 23:04:16.959471 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:16.958906 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" podUID="cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" containerName="maas-api" containerID="cri-o://b3b515e8556e1e60fece1606aeb21272c87c1a071c69d7dd77a40e9042fbb80a" gracePeriod=30 Nov 03 23:04:17.198412 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.198069 4898 generic.go:356] "Generic (PLEG): container finished" podID="cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" containerID="b3b515e8556e1e60fece1606aeb21272c87c1a071c69d7dd77a40e9042fbb80a" exitCode=0 Nov 03 23:04:17.198412 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.198229 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" event={"ID":"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9","Type":"ContainerDied","Data":"b3b515e8556e1e60fece1606aeb21272c87c1a071c69d7dd77a40e9042fbb80a"} Nov 03 23:04:17.294610 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.294572 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 23:04:17.398046 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.398011 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls\") pod \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " Nov 03 23:04:17.398266 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.398098 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-kube-api-access-f8l7q\" (UniqueName: \"kubernetes.io/projected/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-kube-api-access-f8l7q\") pod \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " Nov 03 23:04:17.398266 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.398216 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-metrics-certs\") pod \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " Nov 03 23:04:17.398266 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.398258 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-openshift-service-ca\") pod \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\" (UID: \"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9\") " Nov 03 23:04:17.398914 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.398884 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-openshift-service-ca" pod "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" (UID: "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:17.401028 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.401002 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-kube-api-access-f8l7q" pod "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" (UID: "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9"). InnerVolumeSpecName "kube-api-access-f8l7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:17.401125 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.401003 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-metrics-certs" pod "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" (UID: "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:17.401125 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.401011 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls" pod "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" (UID: "cc8494eb-5c7c-4df2-886a-eb4f8e4385b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:17.499647 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.499609 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:17.499647 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.499640 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8l7q\" (UniqueName: \"kubernetes.io/projected/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-kube-api-access-f8l7q\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:17.499647 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.499651 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:17.500021 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:17.499660 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:18.205370 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:18.205336 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" Nov 03 23:04:18.205852 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:18.205372 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" event={"ID":"cc8494eb-5c7c-4df2-886a-eb4f8e4385b9","Type":"ContainerDied","Data":"2030c88af75dcbd1a0a7416371e498b64feca74132c2de1de930aa247c6c43f2"} Nov 03 23:04:18.205852 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:18.205433 4898 scope.go:122] "RemoveContainer" containerID="b3b515e8556e1e60fece1606aeb21272c87c1a071c69d7dd77a40e9042fbb80a" Nov 03 23:04:18.371488 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:18.371456 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz"] Nov 03 23:04:18.421855 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:18.421813 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz"] Nov 03 23:04:18.909629 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:18.909591 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" path="/var/lib/kubelet/pods/cc8494eb-5c7c-4df2-886a-eb4f8e4385b9/volumes" Nov 03 23:04:19.834676 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:04:19.834631 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:04:20.208651 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:20.208563 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18c03a-796657cb79-wkvqz" podUID="cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.55:8443/health\": context deadline exceeded" Nov 03 23:04:32.912299 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:32.912263 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k"] Nov 03 23:04:32.912723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:32.912623 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" podUID="f5006375-cd5b-4d8e-832b-da705da48540" containerName="maas-api" containerID="cri-o://8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40" gracePeriod=30 Nov 03 23:04:33.227008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.226976 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 23:04:33.297515 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.297472 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls\") pod \"f5006375-cd5b-4d8e-832b-da705da48540\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " Nov 03 23:04:33.297725 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.297534 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-metrics-certs\") pod \"f5006375-cd5b-4d8e-832b-da705da48540\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " Nov 03 23:04:33.297725 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.297609 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5006375-cd5b-4d8e-832b-da705da48540-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5006375-cd5b-4d8e-832b-da705da48540-openshift-service-ca\") pod \"f5006375-cd5b-4d8e-832b-da705da48540\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " Nov 03 23:04:33.297725 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.297640 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5006375-cd5b-4d8e-832b-da705da48540-kube-api-access-nsrsg\" (UniqueName: \"kubernetes.io/projected/f5006375-cd5b-4d8e-832b-da705da48540-kube-api-access-nsrsg\") pod \"f5006375-cd5b-4d8e-832b-da705da48540\" (UID: \"f5006375-cd5b-4d8e-832b-da705da48540\") " Nov 03 23:04:33.298977 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.298925 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5006375-cd5b-4d8e-832b-da705da48540-openshift-service-ca" pod "f5006375-cd5b-4d8e-832b-da705da48540" (UID: "f5006375-cd5b-4d8e-832b-da705da48540"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:33.302338 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.302296 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5006375-cd5b-4d8e-832b-da705da48540-kube-api-access-nsrsg" pod "f5006375-cd5b-4d8e-832b-da705da48540" (UID: "f5006375-cd5b-4d8e-832b-da705da48540"). InnerVolumeSpecName "kube-api-access-nsrsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:33.302886 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.302845 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls" pod "f5006375-cd5b-4d8e-832b-da705da48540" (UID: "f5006375-cd5b-4d8e-832b-da705da48540"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:33.314910 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.310362 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-metrics-certs" pod "f5006375-cd5b-4d8e-832b-da705da48540" (UID: "f5006375-cd5b-4d8e-832b-da705da48540"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:33.322581 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.322543 4898 generic.go:356] "Generic (PLEG): container finished" podID="f5006375-cd5b-4d8e-832b-da705da48540" containerID="8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40" exitCode=0 Nov 03 23:04:33.322743 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.322611 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" Nov 03 23:04:33.322743 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.322625 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" event={"ID":"f5006375-cd5b-4d8e-832b-da705da48540","Type":"ContainerDied","Data":"8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40"} Nov 03 23:04:33.322743 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.322693 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k" event={"ID":"f5006375-cd5b-4d8e-832b-da705da48540","Type":"ContainerDied","Data":"d2d03257268992ca3102f95e0844f91b95e7d8d4b4204fb235a44f6994203d29"} Nov 03 23:04:33.322743 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.322722 4898 scope.go:122] "RemoveContainer" containerID="8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40" Nov 03 23:04:33.338127 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.338106 4898 scope.go:122] "RemoveContainer" containerID="8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40" Nov 03 23:04:33.338487 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:04:33.338463 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40\": container with ID starting with 8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40 not found: ID does not exist" containerID="8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40" Nov 03 23:04:33.338583 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.338495 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40"} err="failed to get container status \"8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40\": rpc error: code = NotFound desc = could not find container \"8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40\": container with ID starting with 8fd24d817b441c93bc34758b4924a14059d03b337e9c7a7c85b2343ff9550a40 not found: ID does not exist" Nov 03 23:04:33.399093 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.399057 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:33.399093 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.399095 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5006375-cd5b-4d8e-832b-da705da48540-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:33.399309 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.399115 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5006375-cd5b-4d8e-832b-da705da48540-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:33.399309 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.399132 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsrsg\" (UniqueName: \"kubernetes.io/projected/f5006375-cd5b-4d8e-832b-da705da48540-kube-api-access-nsrsg\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:33.477888 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.463100 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k"] Nov 03 23:04:33.482502 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:33.482466 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-31ad7b-6db975795c-mfp7k"] Nov 03 23:04:34.906592 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:34.906555 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5006375-cd5b-4d8e-832b-da705da48540" path="/var/lib/kubelet/pods/f5006375-cd5b-4d8e-832b-da705da48540/volumes" Nov 03 23:04:42.546964 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:42.546916 4898 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal" podUID="3dade66e2d9f1769c6a8531aeeaf63ad" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:49.840430 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:04:49.840382 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:04:51.572287 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.572247 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792"] Nov 03 23:04:51.572983 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.572966 4898 cpu_manager.go:414] "containerMap: removing container" podUID="0404aee4-fa58-4d0d-9eac-63ac403503d5" containerName="payload-pre-processing" Nov 03 23:04:51.573047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.572985 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0404aee4-fa58-4d0d-9eac-63ac403503d5" containerName="payload-pre-processing" Nov 03 23:04:51.573047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573003 4898 cpu_manager.go:414] "containerMap: removing container" podUID="f5006375-cd5b-4d8e-832b-da705da48540" containerName="maas-api" Nov 03 23:04:51.573047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573011 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5006375-cd5b-4d8e-832b-da705da48540" containerName="maas-api" Nov 03 23:04:51.573047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573029 4898 cpu_manager.go:414] "containerMap: removing container" podUID="670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" containerName="maas-api" Nov 03 23:04:51.573047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573036 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" containerName="maas-api" Nov 03 23:04:51.573047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573044 4898 cpu_manager.go:414] "containerMap: removing container" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573051 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573066 4898 cpu_manager.go:414] "containerMap: removing container" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573073 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573086 4898 cpu_manager.go:414] "containerMap: removing container" podUID="cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" containerName="maas-api" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573092 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" containerName="maas-api" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573118 4898 cpu_manager.go:414] "containerMap: removing container" podUID="06e66f7f-49d3-4e1d-9557-b8c57159c381" containerName="payload-processing" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573127 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e66f7f-49d3-4e1d-9557-b8c57159c381" containerName="payload-processing" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573149 4898 cpu_manager.go:414] "containerMap: removing container" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573155 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573176 4898 cpu_manager.go:414] "containerMap: removing container" podUID="82a88795-e98d-4780-b672-3a6dee25b4d3" containerName="maas-api" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573182 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a88795-e98d-4780-b672-3a6dee25b4d3" containerName="maas-api" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573193 4898 cpu_manager.go:414] "containerMap: removing container" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573199 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573213 4898 cpu_manager.go:414] "containerMap: removing container" podUID="ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" containerName="payload-pre-processing" Nov 03 23:04:51.573229 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573220 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" containerName="payload-pre-processing" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573319 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f23050-48f9-4b70-b3e5-0794b749e533" containerName="main" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573331 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5068bb-6e95-43f3-a4ef-e7f27fdca4da" containerName="payload-pre-processing" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573345 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0404aee4-fa58-4d0d-9eac-63ac403503d5" containerName="payload-pre-processing" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573355 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5006375-cd5b-4d8e-832b-da705da48540" containerName="maas-api" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573364 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="670a3d97-b6f5-4d5f-9d8c-5c1e678d0cfa" containerName="maas-api" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573373 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4f1c63-7c5f-4df2-a48e-b1c9204d313c" containerName="istio-proxy" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573383 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573392 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8494eb-5c7c-4df2-886a-eb4f8e4385b9" containerName="maas-api" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573403 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e66f7f-49d3-4e1d-9557-b8c57159c381" containerName="payload-processing" Nov 03 23:04:51.573673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.573417 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a88795-e98d-4780-b672-3a6dee25b4d3" containerName="maas-api" Nov 03 23:04:51.581478 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.581455 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.602920 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.602884 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-model-cache\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.603100 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.603002 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czt8k\" (UniqueName: \"kubernetes.io/projected/953b0740-f330-4f73-a1b6-a1f9718acd7c-kube-api-access-czt8k\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.603100 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.603069 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-dshm\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.603239 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.603099 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-home\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.603239 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.603183 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-tmp-dir\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.603350 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.603290 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953b0740-f330-4f73-a1b6-a1f9718acd7c-tls-certs\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.634902 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.634865 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792"] Nov 03 23:04:51.680514 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.680477 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0bf80c\"/\"default-dockercfg-h4qb4\"" Nov 03 23:04:51.680713 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.680565 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0bf80c\"/\"test-model-0bf80c-kserve-self-signed-certs\"" Nov 03 23:04:51.684866 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.684825 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0bf80c\"/\"kube-root-ca.crt\"" Nov 03 23:04:51.685013 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.684956 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0bf80c\"/\"openshift-service-ca.crt\"" Nov 03 23:04:51.705004 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.704966 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953b0740-f330-4f73-a1b6-a1f9718acd7c-tls-certs\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.706545 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.706513 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-model-cache\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.706820 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.706591 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czt8k\" (UniqueName: \"kubernetes.io/projected/953b0740-f330-4f73-a1b6-a1f9718acd7c-kube-api-access-czt8k\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.707541 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.707114 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-model-cache\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.707541 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.707234 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-dshm\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.707541 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.707268 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-home\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.707541 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.707304 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-tmp-dir\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.707950 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.707702 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-tmp-dir\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.708187 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.708165 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-home\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.710427 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.710406 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953b0740-f330-4f73-a1b6-a1f9718acd7c-tls-certs\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.713308 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.713255 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-dshm\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.773104 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.773073 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czt8k\" (UniqueName: \"kubernetes.io/projected/953b0740-f330-4f73-a1b6-a1f9718acd7c-kube-api-access-czt8k\") pod \"test-model-0bf80c-kserve-5987f9668f-wd792\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:51.900957 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:51.900873 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:04:52.287223 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:52.287194 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792"] Nov 03 23:04:52.288541 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:04:52.288511 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod953b0740_f330_4f73_a1b6_a1f9718acd7c.slice/crio-550d383336826df05f6562b9d3f29b6b7313e9499503fb76fa95511da640fa9e WatchSource:0}: Error finding container 550d383336826df05f6562b9d3f29b6b7313e9499503fb76fa95511da640fa9e: Status 404 returned error can't find the container with id 550d383336826df05f6562b9d3f29b6b7313e9499503fb76fa95511da640fa9e Nov 03 23:04:52.443255 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:52.443216 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" event={"ID":"953b0740-f330-4f73-a1b6-a1f9718acd7c","Type":"ContainerStarted","Data":"fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657"} Nov 03 23:04:52.443255 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:52.443261 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" event={"ID":"953b0740-f330-4f73-a1b6-a1f9718acd7c","Type":"ContainerStarted","Data":"550d383336826df05f6562b9d3f29b6b7313e9499503fb76fa95511da640fa9e"} Nov 03 23:04:52.561617 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:52.561507 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" podStartSLOduration=1.561492034 podStartE2EDuration="1.561492034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:52.559905879 +0000 UTC m=+2535.772674915" watchObservedRunningTime="2026-11-03 23:04:52.561492034 +0000 UTC m=+2535.774261085" Nov 03 23:04:52.914245 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:04:52.914211 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:05:01.901613 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:05:01.901578 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:05:01.914683 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:05:01.914653 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:05:02.516604 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:05:02.516574 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:05:11.052274 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:05:11.052237 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr"] Nov 03 23:05:11.102906 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:05:11.102849 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a47cc74-dwzqr"] Nov 03 23:05:12.906507 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:05:12.906473 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cccc36dc-6067-4498-a005-2a256c0069c2" path="/var/lib/kubelet/pods/cccc36dc-6067-4498-a005-2a256c0069c2/volumes" Nov 03 23:05:19.835227 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:05:19.835139 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:05:49.836903 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:05:49.836860 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:06:02.159882 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.159835 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs"] Nov 03 23:06:02.160533 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.160119 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" podUID="e6a938fa-5b52-4d3c-bd23-1497ec55fd84" containerName="maas-api" containerID="cri-o://6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253" gracePeriod=30 Nov 03 23:06:02.465131 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.465099 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:06:02.518954 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.518918 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-openshift-service-ca\") pod \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " Nov 03 23:06:02.519138 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.518980 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls\") pod \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " Nov 03 23:06:02.519138 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.519046 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-metrics-certs\") pod \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " Nov 03 23:06:02.519138 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.519097 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-kube-api-access-gvszf\" (UniqueName: \"kubernetes.io/projected/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-kube-api-access-gvszf\") pod \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\" (UID: \"e6a938fa-5b52-4d3c-bd23-1497ec55fd84\") " Nov 03 23:06:02.520354 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.520283 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-openshift-service-ca" pod "e6a938fa-5b52-4d3c-bd23-1497ec55fd84" (UID: "e6a938fa-5b52-4d3c-bd23-1497ec55fd84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:02.522003 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.521964 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-metrics-certs" pod "e6a938fa-5b52-4d3c-bd23-1497ec55fd84" (UID: "e6a938fa-5b52-4d3c-bd23-1497ec55fd84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:02.522131 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.522065 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls" pod "e6a938fa-5b52-4d3c-bd23-1497ec55fd84" (UID: "e6a938fa-5b52-4d3c-bd23-1497ec55fd84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:02.523045 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.523016 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-kube-api-access-gvszf" pod "e6a938fa-5b52-4d3c-bd23-1497ec55fd84" (UID: "e6a938fa-5b52-4d3c-bd23-1497ec55fd84"). InnerVolumeSpecName "kube-api-access-gvszf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:02.620599 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.620566 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:02.620599 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.620599 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvszf\" (UniqueName: \"kubernetes.io/projected/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-kube-api-access-gvszf\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:02.620817 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.620611 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:02.620817 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.620620 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a938fa-5b52-4d3c-bd23-1497ec55fd84-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:02.866255 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.866218 4898 generic.go:356] "Generic (PLEG): container finished" podID="e6a938fa-5b52-4d3c-bd23-1497ec55fd84" containerID="6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253" exitCode=0 Nov 03 23:06:02.866447 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.866270 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" Nov 03 23:06:02.866447 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.866311 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" event={"ID":"e6a938fa-5b52-4d3c-bd23-1497ec55fd84","Type":"ContainerDied","Data":"6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253"} Nov 03 23:06:02.866447 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.866368 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs" event={"ID":"e6a938fa-5b52-4d3c-bd23-1497ec55fd84","Type":"ContainerDied","Data":"0f33462b757bebeb223e47ab96880213fe27648748737e97c3788dca3a5fe5d6"} Nov 03 23:06:02.866447 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.866394 4898 scope.go:122] "RemoveContainer" containerID="6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253" Nov 03 23:06:02.877181 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.877164 4898 scope.go:122] "RemoveContainer" containerID="6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253" Nov 03 23:06:02.877471 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:06:02.877451 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253\": container with ID starting with 6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253 not found: ID does not exist" containerID="6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253" Nov 03 23:06:02.877536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:02.877481 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253"} err="failed to get container status \"6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253\": rpc error: code = NotFound desc = could not find container \"6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253\": container with ID starting with 6fcdc7ac862e5989c51909cd847d1687374c974680fc126179c5c132d116a253 not found: ID does not exist" Nov 03 23:06:03.032306 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:03.032271 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs"] Nov 03 23:06:03.083685 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:03.083649 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-656ee6-5c4f6488cc-229vs"] Nov 03 23:06:04.906368 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:04.906323 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a938fa-5b52-4d3c-bd23-1497ec55fd84" path="/var/lib/kubelet/pods/e6a938fa-5b52-4d3c-bd23-1497ec55fd84/volumes" Nov 03 23:06:11.723169 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:11.723120 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj"] Nov 03 23:06:11.723668 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:11.723391 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" containerID="cri-o://de63c105240dea39795b11dfd8d626feb5f9b8d76951a9fd672f699a388656e6" gracePeriod=30 Nov 03 23:06:16.960213 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:16.952745 4898 generic.go:356] "Generic (PLEG): container finished" podID="c9b11632-5041-429c-aabf-23b9123082e2" containerID="de63c105240dea39795b11dfd8d626feb5f9b8d76951a9fd672f699a388656e6" exitCode=0 Nov 03 23:06:16.960213 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:16.952832 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" event={"ID":"c9b11632-5041-429c-aabf-23b9123082e2","Type":"ContainerDied","Data":"de63c105240dea39795b11dfd8d626feb5f9b8d76951a9fd672f699a388656e6"} Nov 03 23:06:16.960213 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:16.952901 4898 scope.go:122] "RemoveContainer" containerID="b18ee4668f759b81194cac1011c272db830a336d4cd538f79d490185814c8c82" Nov 03 23:06:17.018999 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.018974 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:06:17.094539 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094506 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-data\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.094712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094591 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-credential-socket\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.094712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094633 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b11632-5041-429c-aabf-23b9123082e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b11632-5041-429c-aabf-23b9123082e2-istiod-ca-cert\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.094712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094702 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-envoy\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.094918 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094736 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-certs\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.094918 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094767 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-data" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:17.094918 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094772 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9b11632-5041-429c-aabf-23b9123082e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b11632-5041-429c-aabf-23b9123082e2-istio-podinfo\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.095077 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094921 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-credential-socket" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:17.095077 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094937 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-kube-api-access-2hqgm\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-kube-api-access-2hqgm\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.095077 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.094991 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-istio-token\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-istio-token\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.095077 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095045 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-socket\") pod \"c9b11632-5041-429c-aabf-23b9123082e2\" (UID: \"c9b11632-5041-429c-aabf-23b9123082e2\") " Nov 03 23:06:17.095261 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095068 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-certs" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:17.095261 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095146 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b11632-5041-429c-aabf-23b9123082e2-istiod-ca-cert" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:17.095432 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095410 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-socket" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:17.095508 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095468 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b11632-5041-429c-aabf-23b9123082e2-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.095508 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095486 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.095508 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095499 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.095508 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.095508 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.097006 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.096980 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9b11632-5041-429c-aabf-23b9123082e2-istio-podinfo" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:17.097127 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.097052 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-istio-token" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:17.097408 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.097390 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-envoy" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:17.097471 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.097421 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-kube-api-access-2hqgm" pod "c9b11632-5041-429c-aabf-23b9123082e2" (UID: "c9b11632-5041-429c-aabf-23b9123082e2"). InnerVolumeSpecName "kube-api-access-2hqgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:17.197378 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.197343 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.197378 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.197376 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b11632-5041-429c-aabf-23b9123082e2-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.197378 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.197388 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b11632-5041-429c-aabf-23b9123082e2-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.197639 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.197401 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqgm\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-kube-api-access-2hqgm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.197639 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.197413 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b11632-5041-429c-aabf-23b9123082e2-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:17.199835 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.199782 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2"] Nov 03 23:06:17.961169 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.961135 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" Nov 03 23:06:17.961669 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.961175 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj" event={"ID":"c9b11632-5041-429c-aabf-23b9123082e2","Type":"ContainerDied","Data":"0b7c8f7c806c54a2d67da0872261b0ed82bdb1f6fc664c83e3c9a2bd7e0fdca1"} Nov 03 23:06:17.961669 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:17.961238 4898 scope.go:122] "RemoveContainer" containerID="de63c105240dea39795b11dfd8d626feb5f9b8d76951a9fd672f699a388656e6" Nov 03 23:06:18.177686 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:18.177653 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj"] Nov 03 23:06:18.227890 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:18.227807 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-656ee6-openshift-default-7bc94fbdf7-lf2vj"] Nov 03 23:06:18.906134 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:18.906094 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b11632-5041-429c-aabf-23b9123082e2" path="/var/lib/kubelet/pods/c9b11632-5041-429c-aabf-23b9123082e2/volumes" Nov 03 23:06:19.834497 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:06:19.834459 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:06:30.204954 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:30.204920 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv"] Nov 03 23:06:32.221240 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.221199 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" containerID="cri-o://4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e" gracePeriod=45 Nov 03 23:06:32.487138 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.487079 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 23:06:32.578638 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.578600 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-home\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-home\") pod \"3d8093ff-1802-42b4-bf19-449e104baf04\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " Nov 03 23:06:32.578846 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.578653 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8093ff-1802-42b4-bf19-449e104baf04-kube-api-access-89prd\" (UniqueName: \"kubernetes.io/projected/3d8093ff-1802-42b4-bf19-449e104baf04-kube-api-access-89prd\") pod \"3d8093ff-1802-42b4-bf19-449e104baf04\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " Nov 03 23:06:32.578846 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.578693 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-tmp-dir\") pod \"3d8093ff-1802-42b4-bf19-449e104baf04\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " Nov 03 23:06:32.578986 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.578899 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8093ff-1802-42b4-bf19-449e104baf04-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d8093ff-1802-42b4-bf19-449e104baf04-tls-certs\") pod \"3d8093ff-1802-42b4-bf19-449e104baf04\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " Nov 03 23:06:32.579044 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.578981 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-tmp-dir" pod "3d8093ff-1802-42b4-bf19-449e104baf04" (UID: "3d8093ff-1802-42b4-bf19-449e104baf04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.579044 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.578990 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-home" pod "3d8093ff-1802-42b4-bf19-449e104baf04" (UID: "3d8093ff-1802-42b4-bf19-449e104baf04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.579044 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.579027 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-dshm\") pod \"3d8093ff-1802-42b4-bf19-449e104baf04\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " Nov 03 23:06:32.579183 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.579057 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-model-cache\") pod \"3d8093ff-1802-42b4-bf19-449e104baf04\" (UID: \"3d8093ff-1802-42b4-bf19-449e104baf04\") " Nov 03 23:06:32.579548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.579491 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.579548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.579516 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.579757 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.579556 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-model-cache" pod "3d8093ff-1802-42b4-bf19-449e104baf04" (UID: "3d8093ff-1802-42b4-bf19-449e104baf04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.581843 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.581816 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8093ff-1802-42b4-bf19-449e104baf04-kube-api-access-89prd" pod "3d8093ff-1802-42b4-bf19-449e104baf04" (UID: "3d8093ff-1802-42b4-bf19-449e104baf04"). InnerVolumeSpecName "kube-api-access-89prd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:32.582015 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.581902 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-dshm" pod "3d8093ff-1802-42b4-bf19-449e104baf04" (UID: "3d8093ff-1802-42b4-bf19-449e104baf04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.582015 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.581968 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8093ff-1802-42b4-bf19-449e104baf04-tls-certs" pod "3d8093ff-1802-42b4-bf19-449e104baf04" (UID: "3d8093ff-1802-42b4-bf19-449e104baf04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:32.681572 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.681543 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d8093ff-1802-42b4-bf19-449e104baf04-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.681829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.681813 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.681949 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.681937 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d8093ff-1802-42b4-bf19-449e104baf04-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.682050 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:32.682038 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89prd\" (UniqueName: \"kubernetes.io/projected/3d8093ff-1802-42b4-bf19-449e104baf04-kube-api-access-89prd\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:33.057437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.056330 4898 generic.go:356] "Generic (PLEG): container finished" podID="3d8093ff-1802-42b4-bf19-449e104baf04" containerID="4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e" exitCode=0 Nov 03 23:06:33.057437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.056515 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" event={"ID":"3d8093ff-1802-42b4-bf19-449e104baf04","Type":"ContainerDied","Data":"4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e"} Nov 03 23:06:33.057437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.056555 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" event={"ID":"3d8093ff-1802-42b4-bf19-449e104baf04","Type":"ContainerDied","Data":"39757cf6de4d229650911a030bcdf8daeef79166ad2fae20814bec2a3715d97c"} Nov 03 23:06:33.057437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.056576 4898 scope.go:122] "RemoveContainer" containerID="4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e" Nov 03 23:06:33.057437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.056770 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2" Nov 03 23:06:33.072328 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.072286 4898 scope.go:122] "RemoveContainer" containerID="4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e" Nov 03 23:06:33.072869 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:06:33.072826 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e\": container with ID starting with 4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e not found: ID does not exist" containerID="4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e" Nov 03 23:06:33.072988 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.072890 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e"} err="failed to get container status \"4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e\": rpc error: code = NotFound desc = could not find container \"4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e\": container with ID starting with 4849ed3cf8ae071d26c36f1c0ecc3838e7d2232403b65fb1de9f9f213d2ba34e not found: ID does not exist" Nov 03 23:06:33.223074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.222986 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2"] Nov 03 23:06:33.273165 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:33.273134 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/premium-w6-5e72ad-kserve-6b8cc5f46b-fmdb2"] Nov 03 23:06:34.905585 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:34.905546 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" path="/var/lib/kubelet/pods/3d8093ff-1802-42b4-bf19-449e104baf04/volumes" Nov 03 23:06:39.570266 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:39.570231 4898 scope.go:122] "RemoveContainer" containerID="79942930f6f4181be66bca4293cdb69fe0a123d89c14c51b8953367ddda52b89" Nov 03 23:06:39.581361 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:39.581336 4898 scope.go:122] "RemoveContainer" containerID="efceb607689d13d6c7bf6d8c03332c27a6a02a5f2ed289c04bc80fa446683c04" Nov 03 23:06:45.225588 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.225546 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" podUID="7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" containerName="main" containerID="cri-o://3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da" gracePeriod=45 Nov 03 23:06:45.510950 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.510921 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:06:45.616754 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.616717 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-home\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-home\") pod \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " Nov 03 23:06:45.616968 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.616901 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tls-certs\") pod \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " Nov 03 23:06:45.616968 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.616943 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tmp-dir\") pod \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " Nov 03 23:06:45.617107 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.616981 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-kube-api-access-jjdc8\" (UniqueName: \"kubernetes.io/projected/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-kube-api-access-jjdc8\") pod \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " Nov 03 23:06:45.617107 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.616978 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-home" pod "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" (UID: "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:45.617107 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.617079 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-model-cache\") pod \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " Nov 03 23:06:45.617276 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.617123 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-dshm\") pod \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\" (UID: \"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11\") " Nov 03 23:06:45.617276 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.617221 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tmp-dir" pod "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" (UID: "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:45.617385 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.617366 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-model-cache" pod "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" (UID: "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:45.617657 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.617593 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:45.617657 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.617621 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:45.617657 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.617635 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:45.619527 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.619498 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-dshm" pod "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" (UID: "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:45.619650 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.619502 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tls-certs" pod "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" (UID: "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:45.619650 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.619581 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-kube-api-access-jjdc8" pod "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" (UID: "7d33dd8f-92b5-4625-8ff0-3595e5d3bd11"). InnerVolumeSpecName "kube-api-access-jjdc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:45.718293 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.718261 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:45.718293 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.718291 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjdc8\" (UniqueName: \"kubernetes.io/projected/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-kube-api-access-jjdc8\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:45.718494 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:45.718302 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:46.141868 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.141834 4898 generic.go:356] "Generic (PLEG): container finished" podID="7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" containerID="3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da" exitCode=0 Nov 03 23:06:46.142087 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.141899 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" Nov 03 23:06:46.142087 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.141921 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" event={"ID":"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11","Type":"ContainerDied","Data":"3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da"} Nov 03 23:06:46.142087 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.141974 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv" event={"ID":"7d33dd8f-92b5-4625-8ff0-3595e5d3bd11","Type":"ContainerDied","Data":"1ca4e3a615fc9d0e1456e74b92cbbe357b89fe93f3489e0079408f5683a5e5ae"} Nov 03 23:06:46.142087 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.141996 4898 scope.go:122] "RemoveContainer" containerID="3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da" Nov 03 23:06:46.152679 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.152657 4898 scope.go:122] "RemoveContainer" containerID="3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da" Nov 03 23:06:46.152998 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:06:46.152979 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da\": container with ID starting with 3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da not found: ID does not exist" containerID="3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da" Nov 03 23:06:46.153093 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.153006 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da"} err="failed to get container status \"3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da\": rpc error: code = NotFound desc = could not find container \"3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da\": container with ID starting with 3682b3cb8c8e39a6fae268c19e9725238fcda27ca495a0f4247e4a8fb8b661da not found: ID does not exist" Nov 03 23:06:46.350640 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.350602 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv"] Nov 03 23:06:46.406984 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.406897 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2cea18/test-model-2cea18-kserve-c8bc5fc8c-tbfzv"] Nov 03 23:06:46.907461 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:46.907385 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" path="/var/lib/kubelet/pods/7d33dd8f-92b5-4625-8ff0-3595e5d3bd11/volumes" Nov 03 23:06:48.469069 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.469034 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq"] Nov 03 23:06:48.469525 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.469262 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" podUID="8d717beb-5c93-4f7c-b3dc-f925dd9ee844" containerName="maas-api" containerID="cri-o://14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad" gracePeriod=30 Nov 03 23:06:48.778984 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.778944 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 23:06:48.850943 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.850051 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-metrics-certs\") pod \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " Nov 03 23:06:48.850943 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.850117 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-openshift-service-ca\") pod \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " Nov 03 23:06:48.850943 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.850151 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls\") pod \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " Nov 03 23:06:48.850943 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.850204 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-kube-api-access-l8xm4\" (UniqueName: \"kubernetes.io/projected/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-kube-api-access-l8xm4\") pod \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\" (UID: \"8d717beb-5c93-4f7c-b3dc-f925dd9ee844\") " Nov 03 23:06:48.851287 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.851064 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-openshift-service-ca" pod "8d717beb-5c93-4f7c-b3dc-f925dd9ee844" (UID: "8d717beb-5c93-4f7c-b3dc-f925dd9ee844"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:48.855517 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.855327 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-metrics-certs" pod "8d717beb-5c93-4f7c-b3dc-f925dd9ee844" (UID: "8d717beb-5c93-4f7c-b3dc-f925dd9ee844"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:48.855517 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.855445 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-kube-api-access-l8xm4" pod "8d717beb-5c93-4f7c-b3dc-f925dd9ee844" (UID: "8d717beb-5c93-4f7c-b3dc-f925dd9ee844"). InnerVolumeSpecName "kube-api-access-l8xm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:48.858405 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.858372 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls" pod "8d717beb-5c93-4f7c-b3dc-f925dd9ee844" (UID: "8d717beb-5c93-4f7c-b3dc-f925dd9ee844"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:48.951694 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.951654 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:48.951694 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.951684 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:48.951694 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.951695 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8xm4\" (UniqueName: \"kubernetes.io/projected/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-kube-api-access-l8xm4\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:48.951694 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:48.951704 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d717beb-5c93-4f7c-b3dc-f925dd9ee844-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:49.162079 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.162045 4898 generic.go:356] "Generic (PLEG): container finished" podID="8d717beb-5c93-4f7c-b3dc-f925dd9ee844" containerID="14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad" exitCode=0 Nov 03 23:06:49.162263 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.162101 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" Nov 03 23:06:49.162263 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.162127 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" event={"ID":"8d717beb-5c93-4f7c-b3dc-f925dd9ee844","Type":"ContainerDied","Data":"14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad"} Nov 03 23:06:49.162263 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.162181 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq" event={"ID":"8d717beb-5c93-4f7c-b3dc-f925dd9ee844","Type":"ContainerDied","Data":"66815e818ef413dc227c8f3f7b8098dcbf64835d15797f04007b6426fdb4fb32"} Nov 03 23:06:49.162263 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.162209 4898 scope.go:122] "RemoveContainer" containerID="14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad" Nov 03 23:06:49.171578 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.171558 4898 scope.go:122] "RemoveContainer" containerID="14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad" Nov 03 23:06:49.171913 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:06:49.171892 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad\": container with ID starting with 14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad not found: ID does not exist" containerID="14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad" Nov 03 23:06:49.172001 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.171921 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad"} err="failed to get container status \"14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad\": rpc error: code = NotFound desc = could not find container \"14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad\": container with ID starting with 14241e7a0acc465bf54680a222eb46837978c346003d463ff0c799edc81704ad not found: ID does not exist" Nov 03 23:06:49.328465 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.328429 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq"] Nov 03 23:06:49.332254 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:49.332228 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b4383-f54f5fc79-j72zq"] Nov 03 23:06:49.848971 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:06:49.848914 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:06:50.906456 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:06:50.906424 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d717beb-5c93-4f7c-b3dc-f925dd9ee844" path="/var/lib/kubelet/pods/8d717beb-5c93-4f7c-b3dc-f925dd9ee844/volumes" Nov 03 23:07:12.177743 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.177706 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht"] Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178217 4898 cpu_manager.go:414] "containerMap: removing container" podUID="7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" containerName="main" Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178231 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" containerName="main" Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178249 4898 cpu_manager.go:414] "containerMap: removing container" podUID="8d717beb-5c93-4f7c-b3dc-f925dd9ee844" containerName="maas-api" Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178255 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d717beb-5c93-4f7c-b3dc-f925dd9ee844" containerName="maas-api" Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178270 4898 cpu_manager.go:414] "containerMap: removing container" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178275 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178283 4898 cpu_manager.go:414] "containerMap: removing container" podUID="e6a938fa-5b52-4d3c-bd23-1497ec55fd84" containerName="maas-api" Nov 03 23:07:12.178291 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178287 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a938fa-5b52-4d3c-bd23-1497ec55fd84" containerName="maas-api" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178309 4898 cpu_manager.go:414] "containerMap: removing container" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178314 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178327 4898 cpu_manager.go:414] "containerMap: removing container" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178331 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178396 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178404 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8093ff-1802-42b4-bf19-449e104baf04" containerName="main" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178410 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d33dd8f-92b5-4625-8ff0-3595e5d3bd11" containerName="main" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178420 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d717beb-5c93-4f7c-b3dc-f925dd9ee844" containerName="maas-api" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178427 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b11632-5041-429c-aabf-23b9123082e2" containerName="istio-proxy" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178432 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a938fa-5b52-4d3c-bd23-1497ec55fd84" containerName="maas-api" Nov 03 23:07:12.178598 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.178438 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d4eff3-033c-4bcf-9fcd-051294b437b3" containerName="istio-proxy" Nov 03 23:07:12.181887 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.181866 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.234944 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.234912 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0bf80c-openshift-default-dockercfg-lcrrz\"" Nov 03 23:07:12.287720 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.287683 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht"] Nov 03 23:07:12.311105 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311076 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-credential-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311311 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311214 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-podinfo\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311311 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311280 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-token\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311311 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311306 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311501 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311472 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-data\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311567 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311503 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-certs\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311567 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311528 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4f2a4d-50bb-4517-be66-a5002ae59735-istiod-ca-cert\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311567 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311552 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxxl\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-kube-api-access-2kxxl\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.311735 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.311649 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-envoy\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.412489 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412452 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-envoy\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.412676 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412499 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-credential-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.412676 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412574 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-podinfo\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.412676 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412627 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-token\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.412676 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412657 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412724 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-data\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412757 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-certs\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412822 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4f2a4d-50bb-4517-be66-a5002ae59735-istiod-ca-cert\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413014 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.412855 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxxl\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-kube-api-access-2kxxl\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413212 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.413021 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-credential-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413212 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.413029 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413342 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.413306 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-certs\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413342 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.413334 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-data\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.413645 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.413624 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4f2a4d-50bb-4517-be66-a5002ae59735-istiod-ca-cert\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.414922 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.414904 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-envoy\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.415168 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.415149 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-podinfo\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.474081 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.474026 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-token\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.474266 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.474248 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxxl\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-kube-api-access-2kxxl\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.495982 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.495959 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:12.928431 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.928399 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht"] Nov 03 23:07:12.930060 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:07:12.930003 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4f2a4d_50bb_4517_be66_a5002ae59735.slice/crio-762ff53b02bfe36f20d04a08b74b6ebeb20afcee7b9e9f205b768fcfa2f64514 WatchSource:0}: Error finding container 762ff53b02bfe36f20d04a08b74b6ebeb20afcee7b9e9f205b768fcfa2f64514: Status 404 returned error can't find the container with id 762ff53b02bfe36f20d04a08b74b6ebeb20afcee7b9e9f205b768fcfa2f64514 Nov 03 23:07:12.933038 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.932997 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:07:12.933131 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.933087 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:07:12.933131 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:12.933120 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:07:13.301351 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:13.301265 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" event={"ID":"eb4f2a4d-50bb-4517-be66-a5002ae59735","Type":"ContainerStarted","Data":"16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883"} Nov 03 23:07:13.301351 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:13.301329 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" event={"ID":"eb4f2a4d-50bb-4517-be66-a5002ae59735","Type":"ContainerStarted","Data":"762ff53b02bfe36f20d04a08b74b6ebeb20afcee7b9e9f205b768fcfa2f64514"} Nov 03 23:07:13.432492 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:13.430176 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podStartSLOduration=1.43015741 podStartE2EDuration="1.43015741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:13.428205929 +0000 UTC m=+2676.640974998" watchObservedRunningTime="2026-11-03 23:07:13.43015741 +0000 UTC m=+2676.642926462" Nov 03 23:07:13.496395 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:13.496357 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:13.498274 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:13.498242 4898 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 23:07:13.498426 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:13.498295 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 23:07:14.496916 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:14.496884 4898 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 23:07:14.497317 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:14.496934 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 23:07:15.496833 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:15.496771 4898 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 23:07:15.496998 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:15.496865 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 23:07:16.496956 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:16.496919 4898 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 23:07:16.497142 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:16.496986 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 23:07:17.496712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:17.496678 4898 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 23:07:17.496901 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:17.496763 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 23:07:18.497169 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:18.497132 4898 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 23:07:18.497575 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:18.497186 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 23:07:19.500478 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:19.500444 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:19.500990 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:19.500906 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:19.501694 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:19.501670 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:07:19.856108 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:07:19.856069 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:07:20.724581 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:20.724546 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn"] Nov 03 23:07:20.725025 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:20.724772 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" podUID="2bf69038-2c53-4767-ab1e-a0447f0af9dd" containerName="maas-api" containerID="cri-o://10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb" gracePeriod=30 Nov 03 23:07:20.997877 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:20.997846 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:07:21.061167 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.061125 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf69038-2c53-4767-ab1e-a0447f0af9dd-kube-api-access-gmjff\" (UniqueName: \"kubernetes.io/projected/2bf69038-2c53-4767-ab1e-a0447f0af9dd-kube-api-access-gmjff\") pod \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " Nov 03 23:07:21.061327 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.061277 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-metrics-certs\") pod \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " Nov 03 23:07:21.061406 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.061390 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf69038-2c53-4767-ab1e-a0447f0af9dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf69038-2c53-4767-ab1e-a0447f0af9dd-openshift-service-ca\") pod \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " Nov 03 23:07:21.061477 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.061456 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls\") pod \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\" (UID: \"2bf69038-2c53-4767-ab1e-a0447f0af9dd\") " Nov 03 23:07:21.061919 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.061886 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf69038-2c53-4767-ab1e-a0447f0af9dd-openshift-service-ca" pod "2bf69038-2c53-4767-ab1e-a0447f0af9dd" (UID: "2bf69038-2c53-4767-ab1e-a0447f0af9dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:21.062375 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.062337 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf69038-2c53-4767-ab1e-a0447f0af9dd-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:21.063706 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.063681 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf69038-2c53-4767-ab1e-a0447f0af9dd-kube-api-access-gmjff" pod "2bf69038-2c53-4767-ab1e-a0447f0af9dd" (UID: "2bf69038-2c53-4767-ab1e-a0447f0af9dd"). InnerVolumeSpecName "kube-api-access-gmjff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:21.063906 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.063876 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-metrics-certs" pod "2bf69038-2c53-4767-ab1e-a0447f0af9dd" (UID: "2bf69038-2c53-4767-ab1e-a0447f0af9dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:21.064371 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.064345 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls" pod "2bf69038-2c53-4767-ab1e-a0447f0af9dd" (UID: "2bf69038-2c53-4767-ab1e-a0447f0af9dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:21.163316 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.163273 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:21.163316 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.163303 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjff\" (UniqueName: \"kubernetes.io/projected/2bf69038-2c53-4767-ab1e-a0447f0af9dd-kube-api-access-gmjff\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:21.163316 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.163315 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf69038-2c53-4767-ab1e-a0447f0af9dd-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:21.348114 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.348074 4898 generic.go:356] "Generic (PLEG): container finished" podID="2bf69038-2c53-4767-ab1e-a0447f0af9dd" containerID="10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb" exitCode=0 Nov 03 23:07:21.348322 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.348140 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" Nov 03 23:07:21.348322 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.348161 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" event={"ID":"2bf69038-2c53-4767-ab1e-a0447f0af9dd","Type":"ContainerDied","Data":"10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb"} Nov 03 23:07:21.348322 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.348256 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn" event={"ID":"2bf69038-2c53-4767-ab1e-a0447f0af9dd","Type":"ContainerDied","Data":"dd29a09b36129b9845e0bf794ec405e2bcdaa6473b2d7ddce76aa744f588bd0d"} Nov 03 23:07:21.348322 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.348286 4898 scope.go:122] "RemoveContainer" containerID="10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb" Nov 03 23:07:21.357948 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.357934 4898 scope.go:122] "RemoveContainer" containerID="10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb" Nov 03 23:07:21.358191 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:07:21.358169 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb\": container with ID starting with 10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb not found: ID does not exist" containerID="10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb" Nov 03 23:07:21.358259 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.358204 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb"} err="failed to get container status \"10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb\": rpc error: code = NotFound desc = could not find container \"10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb\": container with ID starting with 10520761cd210579eac83cb54ed752f3c74b992426367528137b10d6535523fb not found: ID does not exist" Nov 03 23:07:21.517061 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.517016 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn"] Nov 03 23:07:21.522306 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:21.522282 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2cea18-864f4f85dc-s9zkn"] Nov 03 23:07:22.906246 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:22.906210 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf69038-2c53-4767-ab1e-a0447f0af9dd" path="/var/lib/kubelet/pods/2bf69038-2c53-4767-ab1e-a0447f0af9dd/volumes" Nov 03 23:07:29.925141 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:29.925106 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf"] Nov 03 23:07:29.925519 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:29.925308 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" containerID="cri-o://c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194" gracePeriod=30 Nov 03 23:07:32.063433 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:32.063400 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh"] Nov 03 23:07:32.063867 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:32.063640 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" containerID="cri-o://3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119" gracePeriod=30 Nov 03 23:07:32.115114 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:32.115078 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc"] Nov 03 23:07:32.115390 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:32.115341 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" containerID="cri-o://3e43701edda68272023e0a9978274003e8efe52f159f3ac7b8bc2d9f3eded88e" gracePeriod=30 Nov 03 23:07:32.118555 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:32.118527 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j"] Nov 03 23:07:32.118861 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:32.118809 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" containerID="cri-o://359e7cdf125bf78cb930d355536146df3daf61b891a17e221387b0099259208e" gracePeriod=30 Nov 03 23:07:35.193095 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.193036 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 23:07:35.322536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322496 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-podinfo\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.322723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322543 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-certs\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.322723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322617 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-envoy\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.322723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322683 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-data\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.322960 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322735 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-credential-socket\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.322960 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322783 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-socket\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.322960 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322864 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-kube-api-access-jr7k2\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-kube-api-access-jr7k2\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.322960 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322807 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-certs" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:35.322960 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322946 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-token\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-token\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.323207 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.322982 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istiod-ca-cert\") pod \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\" (UID: \"909a0d33-8e64-42a5-9ed0-67cb0a96a413\") " Nov 03 23:07:35.323207 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323018 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-data" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:35.323207 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323059 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-credential-socket" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:35.323207 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323071 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-socket" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:35.323458 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323437 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istiod-ca-cert" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:35.323529 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323505 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.323529 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323529 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.323734 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323543 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.323734 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.323556 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.324951 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.324925 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-podinfo" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:35.325076 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.325058 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-token" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:35.325242 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.325220 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-envoy" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:35.325317 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.325246 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-kube-api-access-jr7k2" pod "909a0d33-8e64-42a5-9ed0-67cb0a96a413" (UID: "909a0d33-8e64-42a5-9ed0-67cb0a96a413"). InnerVolumeSpecName "kube-api-access-jr7k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:35.425390 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.425358 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr7k2\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-kube-api-access-jr7k2\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.425390 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.425389 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.425390 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.425405 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.425646 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.425417 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.425646 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.425431 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a0d33-8e64-42a5-9ed0-67cb0a96a413-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.429339 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.429307 4898 generic.go:356] "Generic (PLEG): container finished" podID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerID="c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194" exitCode=0 Nov 03 23:07:35.429479 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.429363 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" Nov 03 23:07:35.429479 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.429394 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" event={"ID":"909a0d33-8e64-42a5-9ed0-67cb0a96a413","Type":"ContainerDied","Data":"c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194"} Nov 03 23:07:35.429479 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.429458 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf" event={"ID":"909a0d33-8e64-42a5-9ed0-67cb0a96a413","Type":"ContainerDied","Data":"61551284e4c17b6066fae6973e4ca63c1de6a77474fc10314ffa9e3d394560f5"} Nov 03 23:07:35.429612 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.429487 4898 scope.go:122] "RemoveContainer" containerID="c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194" Nov 03 23:07:35.440567 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.440548 4898 scope.go:122] "RemoveContainer" containerID="1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f" Nov 03 23:07:35.449062 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.449042 4898 scope.go:122] "RemoveContainer" containerID="c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194" Nov 03 23:07:35.449368 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:07:35.449344 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194\": container with ID starting with c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194 not found: ID does not exist" containerID="c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194" Nov 03 23:07:35.449434 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.449382 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194"} err="failed to get container status \"c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194\": rpc error: code = NotFound desc = could not find container \"c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194\": container with ID starting with c569130bfc74139564fee862a0baad43e37ac3b7b3ee7d510298d877622a0194 not found: ID does not exist" Nov 03 23:07:35.449434 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.449410 4898 scope.go:122] "RemoveContainer" containerID="1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f" Nov 03 23:07:35.449851 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:07:35.449823 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f\": container with ID starting with 1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f not found: ID does not exist" containerID="1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f" Nov 03 23:07:35.450243 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.449861 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f"} err="failed to get container status \"1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f\": rpc error: code = NotFound desc = could not find container \"1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f\": container with ID starting with 1c3cc275be7788c525c21dd68d39e28e3b449892f00779162b5fcea13800188f not found: ID does not exist" Nov 03 23:07:35.599194 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.599165 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf"] Nov 03 23:07:35.609024 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:35.608997 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2cea18-openshift-default-794d55d96f-rmqhf"] Nov 03 23:07:36.906406 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:36.906375 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" path="/var/lib/kubelet/pods/909a0d33-8e64-42a5-9ed0-67cb0a96a413/volumes" Nov 03 23:07:37.396806 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.396755 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 23:07:37.446405 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.446372 4898 generic.go:356] "Generic (PLEG): container finished" podID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerID="3e43701edda68272023e0a9978274003e8efe52f159f3ac7b8bc2d9f3eded88e" exitCode=0 Nov 03 23:07:37.446519 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.446416 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" event={"ID":"24779e64-aabc-4fa5-8701-79d557ea9c32","Type":"ContainerDied","Data":"3e43701edda68272023e0a9978274003e8efe52f159f3ac7b8bc2d9f3eded88e"} Nov 03 23:07:37.448632 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.448603 4898 generic.go:356] "Generic (PLEG): container finished" podID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerID="359e7cdf125bf78cb930d355536146df3daf61b891a17e221387b0099259208e" exitCode=0 Nov 03 23:07:37.448766 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.448679 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" event={"ID":"78915b29-4a76-4c77-8735-e67fa51c3fc4","Type":"ContainerDied","Data":"359e7cdf125bf78cb930d355536146df3daf61b891a17e221387b0099259208e"} Nov 03 23:07:37.450538 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.450509 4898 generic.go:356] "Generic (PLEG): container finished" podID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerID="3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119" exitCode=0 Nov 03 23:07:37.450654 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.450566 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" Nov 03 23:07:37.450654 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.450590 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" event={"ID":"85456a7b-db98-4291-b3ed-e01fa8d7052f","Type":"ContainerDied","Data":"3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119"} Nov 03 23:07:37.450777 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.450680 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh" event={"ID":"85456a7b-db98-4291-b3ed-e01fa8d7052f","Type":"ContainerDied","Data":"73d2318ebd7311f98da1067c5931cb25b00c94d591a2957fb18f27f12875596c"} Nov 03 23:07:37.450777 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.450709 4898 scope.go:122] "RemoveContainer" containerID="3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119" Nov 03 23:07:37.455168 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.455149 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 23:07:37.460643 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.460619 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 23:07:37.461467 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.461454 4898 scope.go:122] "RemoveContainer" containerID="3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119" Nov 03 23:07:37.461697 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:07:37.461681 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119\": container with ID starting with 3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119 not found: ID does not exist" containerID="3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119" Nov 03 23:07:37.461754 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.461705 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119"} err="failed to get container status \"3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119\": rpc error: code = NotFound desc = could not find container \"3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119\": container with ID starting with 3770c857c8a18173c95b9fcc04685c4183fade3bee54faaa08dc1e0160cc7119 not found: ID does not exist" Nov 03 23:07:37.542034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.542007 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:07:37.546756 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.546730 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:07:37.547710 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547685 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85456a7b-db98-4291-b3ed-e01fa8d7052f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85456a7b-db98-4291-b3ed-e01fa8d7052f-istiod-ca-cert\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.547829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547727 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-certs\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.547829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547759 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-data\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.547829 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547819 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-certs\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547843 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-kube-api-access-75gln\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-kube-api-access-75gln\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.548034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547862 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-envoy\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547904 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-token\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-token\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.548034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547957 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-podinfo\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.547983 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-data\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.548034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548000 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-kube-api-access-pfbfw\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-kube-api-access-pfbfw\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548018 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-envoy\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548032 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-certs" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548045 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-token\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-token\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548055 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85456a7b-db98-4291-b3ed-e01fa8d7052f-istiod-ca-cert" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548066 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-envoy\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548091 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-credential-socket\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548109 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-podinfo\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548125 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-data\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548144 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-podinfo\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548169 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-token\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-token\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548191 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-socket\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548226 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78915b29-4a76-4c77-8735-e67fa51c3fc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78915b29-4a76-4c77-8735-e67fa51c3fc4-istiod-ca-cert\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548250 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-socket\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548280 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-credential-socket\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548299 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24779e64-aabc-4fa5-8701-79d557ea9c32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24779e64-aabc-4fa5-8701-79d557ea9c32-istiod-ca-cert\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548340 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-certs\") pod \"85456a7b-db98-4291-b3ed-e01fa8d7052f\" (UID: \"85456a7b-db98-4291-b3ed-e01fa8d7052f\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548361 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-kube-api-access-dscft\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-kube-api-access-dscft\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.548399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548400 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-socket\") pod \"24779e64-aabc-4fa5-8701-79d557ea9c32\" (UID: \"24779e64-aabc-4fa5-8701-79d557ea9c32\") " Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548417 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-credential-socket\") pod \"78915b29-4a76-4c77-8735-e67fa51c3fc4\" (UID: \"78915b29-4a76-4c77-8735-e67fa51c3fc4\") " Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548433 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-certs" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548618 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-data" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548753 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85456a7b-db98-4291-b3ed-e01fa8d7052f-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548833 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548851 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548867 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.549294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.548849 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-credential-socket" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.550060 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.549542 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-credential-socket" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.550060 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.550018 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-credential-socket" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.550997 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.550536 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24779e64-aabc-4fa5-8701-79d557ea9c32-istiod-ca-cert" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:37.550997 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.550853 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-socket" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.551154 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551004 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-token" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.551154 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551062 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-socket" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.551154 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551082 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-data" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.551339 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551207 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-data" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.551339 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551231 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78915b29-4a76-4c77-8735-e67fa51c3fc4-istiod-ca-cert" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:37.551451 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551410 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-certs" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.551885 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551851 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-podinfo" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:37.551986 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551925 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-podinfo" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:37.551986 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551396 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-socket" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.552097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.551981 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-kube-api-access-75gln" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "kube-api-access-75gln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.552499 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.552464 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-envoy" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.553419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.553372 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-token" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.553419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.553364 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-podinfo" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:37.553930 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.553898 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-token" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.554074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.554039 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-envoy" pod "85456a7b-db98-4291-b3ed-e01fa8d7052f" (UID: "85456a7b-db98-4291-b3ed-e01fa8d7052f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.554646 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.554625 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-kube-api-access-dscft" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "kube-api-access-dscft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.554739 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.554641 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-kube-api-access-pfbfw" pod "78915b29-4a76-4c77-8735-e67fa51c3fc4" (UID: "78915b29-4a76-4c77-8735-e67fa51c3fc4"). InnerVolumeSpecName "kube-api-access-pfbfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.554955 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.554938 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-envoy" pod "24779e64-aabc-4fa5-8701-79d557ea9c32" (UID: "24779e64-aabc-4fa5-8701-79d557ea9c32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.624911 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.624883 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:07:37.629426 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.629399 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:07:37.649629 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649598 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649629 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649627 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649641 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfbfw\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-kube-api-access-pfbfw\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649654 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649668 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649681 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649693 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649705 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24779e64-aabc-4fa5-8701-79d557ea9c32-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649718 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649731 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649743 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649757 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649772 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78915b29-4a76-4c77-8735-e67fa51c3fc4-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649817 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649832 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649846 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24779e64-aabc-4fa5-8701-79d557ea9c32-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649858 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85456a7b-db98-4291-b3ed-e01fa8d7052f-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649871 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscft\" (UniqueName: \"kubernetes.io/projected/24779e64-aabc-4fa5-8701-79d557ea9c32-kube-api-access-dscft\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649884 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24779e64-aabc-4fa5-8701-79d557ea9c32-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649896 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649909 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75gln\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-kube-api-access-75gln\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649922 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78915b29-4a76-4c77-8735-e67fa51c3fc4-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.649972 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.649937 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85456a7b-db98-4291-b3ed-e01fa8d7052f-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.979294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.979254 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh"] Nov 03 23:07:37.982664 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:37.982639 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nr7zh"] Nov 03 23:07:38.380712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.380615 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd"] Nov 03 23:07:38.380968 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.380906 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" containerID="cri-o://be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c" gracePeriod=30 Nov 03 23:07:38.458578 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.458550 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" Nov 03 23:07:38.458775 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.458581 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j" event={"ID":"78915b29-4a76-4c77-8735-e67fa51c3fc4","Type":"ContainerDied","Data":"d2b184991139b5e9abaef678fdbdef3e4b9e6e8de3d1e241e682fda668039f9b"} Nov 03 23:07:38.458775 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.458653 4898 scope.go:122] "RemoveContainer" containerID="359e7cdf125bf78cb930d355536146df3daf61b891a17e221387b0099259208e" Nov 03 23:07:38.461386 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.461357 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" event={"ID":"24779e64-aabc-4fa5-8701-79d557ea9c32","Type":"ContainerDied","Data":"f7594f989b2162b0ef1bbefcda96d78223b33b3c3d85907bbf3f70f5c26a4d89"} Nov 03 23:07:38.461386 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.461380 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc" Nov 03 23:07:38.472332 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.472306 4898 scope.go:122] "RemoveContainer" containerID="3e43701edda68272023e0a9978274003e8efe52f159f3ac7b8bc2d9f3eded88e" Nov 03 23:07:38.680883 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.680783 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc"] Nov 03 23:07:38.687214 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.687186 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jlhrc"] Nov 03 23:07:38.797568 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.797531 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j"] Nov 03 23:07:38.802007 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.801978 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-std4j"] Nov 03 23:07:38.906058 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.906018 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" path="/var/lib/kubelet/pods/24779e64-aabc-4fa5-8701-79d557ea9c32/volumes" Nov 03 23:07:38.906533 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.906512 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" path="/var/lib/kubelet/pods/78915b29-4a76-4c77-8735-e67fa51c3fc4/volumes" Nov 03 23:07:38.906987 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:38.906969 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" path="/var/lib/kubelet/pods/85456a7b-db98-4291-b3ed-e01fa8d7052f/volumes" Nov 03 23:07:43.144284 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.144243 4898 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:07:43.146828 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.144301 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:07:43.648064 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.647998 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 23:07:43.812401 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812364 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-credential-socket\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812587 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812414 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-token\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-token\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812587 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812435 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-kube-api-access-662d4\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-kube-api-access-662d4\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812587 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812459 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-certs\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812769 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812584 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istiod-ca-cert\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812769 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812711 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-data\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812769 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812685 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-credential-socket" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.812955 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812771 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-certs" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.812955 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812839 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-envoy\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812955 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812881 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-socket\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.812955 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812918 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-podinfo\") pod \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\" (UID: \"b86c6c87-ccf4-433e-bf67-850af6e8ec57\") " Nov 03 23:07:43.813172 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.812947 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-data" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.813172 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.813018 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istiod-ca-cert" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:43.813285 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.813187 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-socket" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.813347 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.813304 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.813347 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.813323 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.813347 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.813337 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.813484 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.813353 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.813484 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.813367 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.814834 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.814782 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-token" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:43.814950 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.814854 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-kube-api-access-662d4" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "kube-api-access-662d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:43.815280 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.815259 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-podinfo" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:43.815360 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.815277 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-envoy" pod "b86c6c87-ccf4-433e-bf67-850af6e8ec57" (UID: "b86c6c87-ccf4-433e-bf67-850af6e8ec57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.914059 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.913992 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.914059 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.914021 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.914059 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.914039 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.914059 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:43.914052 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662d4\" (UniqueName: \"kubernetes.io/projected/b86c6c87-ccf4-433e-bf67-850af6e8ec57-kube-api-access-662d4\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:44.498181 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.498145 4898 generic.go:356] "Generic (PLEG): container finished" podID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerID="be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c" exitCode=0 Nov 03 23:07:44.498612 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.498194 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" Nov 03 23:07:44.498612 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.498217 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" event={"ID":"b86c6c87-ccf4-433e-bf67-850af6e8ec57","Type":"ContainerDied","Data":"be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c"} Nov 03 23:07:44.498612 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.498258 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd" event={"ID":"b86c6c87-ccf4-433e-bf67-850af6e8ec57","Type":"ContainerDied","Data":"aff0d3169f6622e28aa5bf45dd024f01ed2ab47b1f143e6b566363dc127c6021"} Nov 03 23:07:44.498612 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.498280 4898 scope.go:122] "RemoveContainer" containerID="be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c" Nov 03 23:07:44.509023 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.508855 4898 scope.go:122] "RemoveContainer" containerID="be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c" Nov 03 23:07:44.509163 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:07:44.509139 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c\": container with ID starting with be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c not found: ID does not exist" containerID="be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c" Nov 03 23:07:44.509224 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.509177 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c"} err="failed to get container status \"be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c\": rpc error: code = NotFound desc = could not find container \"be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c\": container with ID starting with be4e9e979af8e56b051d2f184b5167e8b5c723f34b080475f4dff6588936395c not found: ID does not exist" Nov 03 23:07:44.714759 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.714727 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd"] Nov 03 23:07:44.721218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.721190 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-q27rd"] Nov 03 23:07:44.906379 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:07:44.906344 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" path="/var/lib/kubelet/pods/b86c6c87-ccf4-433e-bf67-850af6e8ec57/volumes" Nov 03 23:07:49.835328 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:07:49.835289 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:08:06.779228 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:06.779191 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x"] Nov 03 23:08:06.779649 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:06.779386 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" podUID="1d915cf7-19ac-4e53-85d5-0b77359cdb93" containerName="maas-api" containerID="cri-o://6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d" gracePeriod=30 Nov 03 23:08:07.064941 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.064915 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 23:08:07.142899 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.142866 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d915cf7-19ac-4e53-85d5-0b77359cdb93-kube-api-access-zgw2s\" (UniqueName: \"kubernetes.io/projected/1d915cf7-19ac-4e53-85d5-0b77359cdb93-kube-api-access-zgw2s\") pod \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " Nov 03 23:08:07.143130 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.142924 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d915cf7-19ac-4e53-85d5-0b77359cdb93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d915cf7-19ac-4e53-85d5-0b77359cdb93-openshift-service-ca\") pod \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " Nov 03 23:08:07.143130 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.142961 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-metrics-certs\") pod \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " Nov 03 23:08:07.143130 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.143066 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-maas-api-tls\") pod \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\" (UID: \"1d915cf7-19ac-4e53-85d5-0b77359cdb93\") " Nov 03 23:08:07.143394 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.143354 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d915cf7-19ac-4e53-85d5-0b77359cdb93-openshift-service-ca" pod "1d915cf7-19ac-4e53-85d5-0b77359cdb93" (UID: "1d915cf7-19ac-4e53-85d5-0b77359cdb93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:07.143532 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.143510 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d915cf7-19ac-4e53-85d5-0b77359cdb93-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:07.145421 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.145390 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d915cf7-19ac-4e53-85d5-0b77359cdb93-kube-api-access-zgw2s" pod "1d915cf7-19ac-4e53-85d5-0b77359cdb93" (UID: "1d915cf7-19ac-4e53-85d5-0b77359cdb93"). InnerVolumeSpecName "kube-api-access-zgw2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:07.145540 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.145441 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-metrics-certs" pod "1d915cf7-19ac-4e53-85d5-0b77359cdb93" (UID: "1d915cf7-19ac-4e53-85d5-0b77359cdb93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:07.145653 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.145631 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-maas-api-tls" pod "1d915cf7-19ac-4e53-85d5-0b77359cdb93" (UID: "1d915cf7-19ac-4e53-85d5-0b77359cdb93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:07.244382 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.244349 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgw2s\" (UniqueName: \"kubernetes.io/projected/1d915cf7-19ac-4e53-85d5-0b77359cdb93-kube-api-access-zgw2s\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:07.244382 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.244378 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:07.244382 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.244388 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d915cf7-19ac-4e53-85d5-0b77359cdb93-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:07.624058 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.624023 4898 generic.go:356] "Generic (PLEG): container finished" podID="1d915cf7-19ac-4e53-85d5-0b77359cdb93" containerID="6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d" exitCode=0 Nov 03 23:08:07.624230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.624075 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" Nov 03 23:08:07.624230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.624097 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" event={"ID":"1d915cf7-19ac-4e53-85d5-0b77359cdb93","Type":"ContainerDied","Data":"6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d"} Nov 03 23:08:07.624230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.624150 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x" event={"ID":"1d915cf7-19ac-4e53-85d5-0b77359cdb93","Type":"ContainerDied","Data":"47c7c3633e015be9c4073745e651c40db50b9760b15fd81b3d468bdeccc8c601"} Nov 03 23:08:07.624230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.624172 4898 scope.go:122] "RemoveContainer" containerID="6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d" Nov 03 23:08:07.634392 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.634375 4898 scope.go:122] "RemoveContainer" containerID="6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d" Nov 03 23:08:07.634690 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:08:07.634666 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d\": container with ID starting with 6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d not found: ID does not exist" containerID="6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d" Nov 03 23:08:07.634818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.634693 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d"} err="failed to get container status \"6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d\": rpc error: code = NotFound desc = could not find container \"6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d\": container with ID starting with 6850548f4b6cb4f41bd44be6fcb79f2fd45603940e4f8dbc310b890b744e782d not found: ID does not exist" Nov 03 23:08:07.790562 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.790524 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x"] Nov 03 23:08:07.794135 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:07.794109 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-68fbd1-7dc4df664d-crx5x"] Nov 03 23:08:08.905960 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:08:08.905929 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d915cf7-19ac-4e53-85d5-0b77359cdb93" path="/var/lib/kubelet/pods/1d915cf7-19ac-4e53-85d5-0b77359cdb93/volumes" Nov 03 23:08:19.834386 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:08:19.834299 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:08:49.834204 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:08:49.834166 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:09:10.493023 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:10.492987 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792"] Nov 03 23:09:19.834330 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:09:19.834285 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:09:25.513031 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.512984 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" podUID="953b0740-f330-4f73-a1b6-a1f9718acd7c" containerName="main" containerID="cri-o://fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657" gracePeriod=45 Nov 03 23:09:25.517491 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.517458 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" podUID="953b0740-f330-4f73-a1b6-a1f9718acd7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": read tcp 10.132.0.2:58478->10.132.0.142:8000: read: connection reset by peer" Nov 03 23:09:25.778377 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.778310 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:09:25.809381 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809346 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-tmp-dir\") pod \"953b0740-f330-4f73-a1b6-a1f9718acd7c\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " Nov 03 23:09:25.809542 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809412 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-model-cache\") pod \"953b0740-f330-4f73-a1b6-a1f9718acd7c\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " Nov 03 23:09:25.809542 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809500 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-dshm\") pod \"953b0740-f330-4f73-a1b6-a1f9718acd7c\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " Nov 03 23:09:25.809542 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809527 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-home\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-home\") pod \"953b0740-f330-4f73-a1b6-a1f9718acd7c\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " Nov 03 23:09:25.809723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809557 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953b0740-f330-4f73-a1b6-a1f9718acd7c-kube-api-access-czt8k\" (UniqueName: \"kubernetes.io/projected/953b0740-f330-4f73-a1b6-a1f9718acd7c-kube-api-access-czt8k\") pod \"953b0740-f330-4f73-a1b6-a1f9718acd7c\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " Nov 03 23:09:25.809723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809576 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/953b0740-f330-4f73-a1b6-a1f9718acd7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/953b0740-f330-4f73-a1b6-a1f9718acd7c-tls-certs\") pod \"953b0740-f330-4f73-a1b6-a1f9718acd7c\" (UID: \"953b0740-f330-4f73-a1b6-a1f9718acd7c\") " Nov 03 23:09:25.809723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809613 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-model-cache" pod "953b0740-f330-4f73-a1b6-a1f9718acd7c" (UID: "953b0740-f330-4f73-a1b6-a1f9718acd7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:25.809723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809624 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-tmp-dir" pod "953b0740-f330-4f73-a1b6-a1f9718acd7c" (UID: "953b0740-f330-4f73-a1b6-a1f9718acd7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:25.809974 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809826 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-home" pod "953b0740-f330-4f73-a1b6-a1f9718acd7c" (UID: "953b0740-f330-4f73-a1b6-a1f9718acd7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:25.809974 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809900 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:25.809974 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.809920 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:25.811746 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.811716 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-dshm" pod "953b0740-f330-4f73-a1b6-a1f9718acd7c" (UID: "953b0740-f330-4f73-a1b6-a1f9718acd7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:25.811746 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.811724 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953b0740-f330-4f73-a1b6-a1f9718acd7c-tls-certs" pod "953b0740-f330-4f73-a1b6-a1f9718acd7c" (UID: "953b0740-f330-4f73-a1b6-a1f9718acd7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:25.811929 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.811772 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953b0740-f330-4f73-a1b6-a1f9718acd7c-kube-api-access-czt8k" pod "953b0740-f330-4f73-a1b6-a1f9718acd7c" (UID: "953b0740-f330-4f73-a1b6-a1f9718acd7c"). InnerVolumeSpecName "kube-api-access-czt8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:25.910760 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.910731 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:25.910760 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.910756 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953b0740-f330-4f73-a1b6-a1f9718acd7c-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:25.910760 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.910766 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czt8k\" (UniqueName: \"kubernetes.io/projected/953b0740-f330-4f73-a1b6-a1f9718acd7c-kube-api-access-czt8k\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:25.911004 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:25.910776 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953b0740-f330-4f73-a1b6-a1f9718acd7c-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:26.032536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.032455 4898 generic.go:356] "Generic (PLEG): container finished" podID="953b0740-f330-4f73-a1b6-a1f9718acd7c" containerID="fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657" exitCode=0 Nov 03 23:09:26.032536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.032504 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" Nov 03 23:09:26.032764 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.032541 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" event={"ID":"953b0740-f330-4f73-a1b6-a1f9718acd7c","Type":"ContainerDied","Data":"fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657"} Nov 03 23:09:26.032764 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.032588 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792" event={"ID":"953b0740-f330-4f73-a1b6-a1f9718acd7c","Type":"ContainerDied","Data":"550d383336826df05f6562b9d3f29b6b7313e9499503fb76fa95511da640fa9e"} Nov 03 23:09:26.032764 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.032610 4898 scope.go:122] "RemoveContainer" containerID="fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657" Nov 03 23:09:26.042860 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.042837 4898 scope.go:122] "RemoveContainer" containerID="fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657" Nov 03 23:09:26.043147 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:09:26.043129 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657\": container with ID starting with fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657 not found: ID does not exist" containerID="fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657" Nov 03 23:09:26.043219 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.043156 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657"} err="failed to get container status \"fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657\": rpc error: code = NotFound desc = could not find container \"fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657\": container with ID starting with fc86496de7fb6af2180a6a07aab6a6b3ce2da40d115fd27edbc869cfaf53b657 not found: ID does not exist" Nov 03 23:09:26.198096 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.198061 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792"] Nov 03 23:09:26.201584 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.201560 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0bf80c/test-model-0bf80c-kserve-5987f9668f-wd792"] Nov 03 23:09:26.905971 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:26.905934 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953b0740-f330-4f73-a1b6-a1f9718acd7c" path="/var/lib/kubelet/pods/953b0740-f330-4f73-a1b6-a1f9718acd7c/volumes" Nov 03 23:09:49.833828 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:09:49.833769 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:09:49.937120 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:49.937084 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz"] Nov 03 23:09:49.937348 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:49.937322 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" podUID="840943c2-edc7-4b62-bb37-c1b5881a743b" containerName="maas-api" containerID="cri-o://384c8f9e98f4ef56945d18b16ebaab65e0174c12be43c88ede638bcb002f3b8f" gracePeriod=30 Nov 03 23:09:50.180351 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.180318 4898 generic.go:356] "Generic (PLEG): container finished" podID="840943c2-edc7-4b62-bb37-c1b5881a743b" containerID="384c8f9e98f4ef56945d18b16ebaab65e0174c12be43c88ede638bcb002f3b8f" exitCode=0 Nov 03 23:09:50.180493 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.180397 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" event={"ID":"840943c2-edc7-4b62-bb37-c1b5881a743b","Type":"ContainerDied","Data":"384c8f9e98f4ef56945d18b16ebaab65e0174c12be43c88ede638bcb002f3b8f"} Nov 03 23:09:50.210398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.210370 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:09:50.354122 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.354086 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840943c2-edc7-4b62-bb37-c1b5881a743b-kube-api-access-s55pb\" (UniqueName: \"kubernetes.io/projected/840943c2-edc7-4b62-bb37-c1b5881a743b-kube-api-access-s55pb\") pod \"840943c2-edc7-4b62-bb37-c1b5881a743b\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " Nov 03 23:09:50.354363 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.354168 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-metrics-certs\") pod \"840943c2-edc7-4b62-bb37-c1b5881a743b\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " Nov 03 23:09:50.354363 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.354213 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/840943c2-edc7-4b62-bb37-c1b5881a743b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840943c2-edc7-4b62-bb37-c1b5881a743b-openshift-service-ca\") pod \"840943c2-edc7-4b62-bb37-c1b5881a743b\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " Nov 03 23:09:50.354363 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.354317 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls\") pod \"840943c2-edc7-4b62-bb37-c1b5881a743b\" (UID: \"840943c2-edc7-4b62-bb37-c1b5881a743b\") " Nov 03 23:09:50.354705 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.354663 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840943c2-edc7-4b62-bb37-c1b5881a743b-openshift-service-ca" pod "840943c2-edc7-4b62-bb37-c1b5881a743b" (UID: "840943c2-edc7-4b62-bb37-c1b5881a743b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:50.356346 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.356313 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840943c2-edc7-4b62-bb37-c1b5881a743b-kube-api-access-s55pb" pod "840943c2-edc7-4b62-bb37-c1b5881a743b" (UID: "840943c2-edc7-4b62-bb37-c1b5881a743b"). InnerVolumeSpecName "kube-api-access-s55pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:50.356831 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.356810 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls" pod "840943c2-edc7-4b62-bb37-c1b5881a743b" (UID: "840943c2-edc7-4b62-bb37-c1b5881a743b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:50.356922 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.356898 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-metrics-certs" pod "840943c2-edc7-4b62-bb37-c1b5881a743b" (UID: "840943c2-edc7-4b62-bb37-c1b5881a743b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:50.455493 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.455459 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:50.455493 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.455490 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840943c2-edc7-4b62-bb37-c1b5881a743b-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:50.455493 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.455502 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840943c2-edc7-4b62-bb37-c1b5881a743b-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:50.455738 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:50.455510 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s55pb\" (UniqueName: \"kubernetes.io/projected/840943c2-edc7-4b62-bb37-c1b5881a743b-kube-api-access-s55pb\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:51.189576 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:51.189489 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" event={"ID":"840943c2-edc7-4b62-bb37-c1b5881a743b","Type":"ContainerDied","Data":"0cae46c2f4f4a935162731a8968822dc0a7b3b737bd90eebfd494bae1b36e37d"} Nov 03 23:09:51.189576 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:51.189522 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz" Nov 03 23:09:51.189576 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:51.189562 4898 scope.go:122] "RemoveContainer" containerID="384c8f9e98f4ef56945d18b16ebaab65e0174c12be43c88ede638bcb002f3b8f" Nov 03 23:09:51.355389 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:51.355351 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz"] Nov 03 23:09:51.405570 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:51.405538 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bf80c-b4ff67b69-tv9wz"] Nov 03 23:09:52.905218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:09:52.905182 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="840943c2-edc7-4b62-bb37-c1b5881a743b" path="/var/lib/kubelet/pods/840943c2-edc7-4b62-bb37-c1b5881a743b/volumes" Nov 03 23:10:12.003326 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:12.003289 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht"] Nov 03 23:10:12.003759 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:12.003512 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" containerID="cri-o://16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883" gracePeriod=30 Nov 03 23:10:17.257818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.257775 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:10:17.300744 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.300671 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-podinfo\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.300927 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.300741 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-certs\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.300927 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.300773 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-kube-api-access-2kxxl\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-kube-api-access-2kxxl\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.300927 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.300817 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4f2a4d-50bb-4517-be66-a5002ae59735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4f2a4d-50bb-4517-be66-a5002ae59735-istiod-ca-cert\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.300927 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.300858 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-credential-socket\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.300927 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.300891 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-socket\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.301200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.300996 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-token\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-token\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.301200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301048 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-data\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.301200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301079 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-envoy\") pod \"eb4f2a4d-50bb-4517-be66-a5002ae59735\" (UID: \"eb4f2a4d-50bb-4517-be66-a5002ae59735\") " Nov 03 23:10:17.301200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301099 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-certs" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.301200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301168 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-credential-socket" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.301450 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301376 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-socket" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.301450 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301398 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-data" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.301536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301467 4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-credential-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.301536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301487 4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.301623 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.301599 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4f2a4d-50bb-4517-be66-a5002ae59735-istiod-ca-cert" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:17.303262 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.303239 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-token" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:17.303262 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.303250 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-envoy" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.303443 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.303412 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-podinfo" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:10:17.303443 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.303412 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-kube-api-access-2kxxl" pod "eb4f2a4d-50bb-4517-be66-a5002ae59735" (UID: "eb4f2a4d-50bb-4517-be66-a5002ae59735"). InnerVolumeSpecName "kube-api-access-2kxxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:17.319300 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.319273 4898 generic.go:356] "Generic (PLEG): container finished" podID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerID="16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883" exitCode=0 Nov 03 23:10:17.319436 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.319335 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" Nov 03 23:10:17.319436 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.319360 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" event={"ID":"eb4f2a4d-50bb-4517-be66-a5002ae59735","Type":"ContainerDied","Data":"16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883"} Nov 03 23:10:17.319436 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.319430 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht" event={"ID":"eb4f2a4d-50bb-4517-be66-a5002ae59735","Type":"ContainerDied","Data":"762ff53b02bfe36f20d04a08b74b6ebeb20afcee7b9e9f205b768fcfa2f64514"} Nov 03 23:10:17.319616 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.319462 4898 scope.go:122] "RemoveContainer" containerID="16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883" Nov 03 23:10:17.329205 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.329184 4898 scope.go:122] "RemoveContainer" containerID="16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883" Nov 03 23:10:17.329461 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:10:17.329442 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883\": container with ID starting with 16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883 not found: ID does not exist" containerID="16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883" Nov 03 23:10:17.329511 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.329470 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883"} err="failed to get container status \"16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883\": rpc error: code = NotFound desc = could not find container \"16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883\": container with ID starting with 16680590a50adacb0b3f0522409d900c9839ff0d845d08267bb0a7ecb5852883 not found: ID does not exist" Nov 03 23:10:17.402198 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.402160 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxxl\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-kube-api-access-2kxxl\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.402198 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.402191 4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4f2a4d-50bb-4517-be66-a5002ae59735-istiod-ca-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.402198 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.402202 4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-workload-socket\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.402446 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.402210 4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-token\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.402446 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.402219 4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-data\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.402446 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.402229 4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-envoy\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.402446 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.402238 4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4f2a4d-50bb-4517-be66-a5002ae59735-istio-podinfo\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.542687 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.542645 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht"] Nov 03 23:10:17.546238 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:17.546208 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-zwtht"] Nov 03 23:10:18.906358 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:18.906319 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" path="/var/lib/kubelet/pods/eb4f2a4d-50bb-4517-be66-a5002ae59735/volumes" Nov 03 23:10:19.836692 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:10:19.836656 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:10:49.765201 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:49.765167 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9"] Nov 03 23:10:49.834189 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:10:49.834142 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:10:56.943440 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943405 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2"] Nov 03 23:10:56.943854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943820 4898 cpu_manager.go:414] "containerMap: removing container" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" Nov 03 23:10:56.943854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943832 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" Nov 03 23:10:56.943854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943845 4898 cpu_manager.go:414] "containerMap: removing container" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" Nov 03 23:10:56.943854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943851 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943866 4898 cpu_manager.go:414] "containerMap: removing container" podUID="1d915cf7-19ac-4e53-85d5-0b77359cdb93" containerName="maas-api" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943871 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d915cf7-19ac-4e53-85d5-0b77359cdb93" containerName="maas-api" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943882 4898 cpu_manager.go:414] "containerMap: removing container" podUID="953b0740-f330-4f73-a1b6-a1f9718acd7c" containerName="main" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943895 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953b0740-f330-4f73-a1b6-a1f9718acd7c" containerName="main" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943912 4898 cpu_manager.go:414] "containerMap: removing container" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943917 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943923 4898 cpu_manager.go:414] "containerMap: removing container" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943927 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943935 4898 cpu_manager.go:414] "containerMap: removing container" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943939 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943947 4898 cpu_manager.go:414] "containerMap: removing container" podUID="2bf69038-2c53-4767-ab1e-a0447f0af9dd" containerName="maas-api" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943951 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf69038-2c53-4767-ab1e-a0447f0af9dd" containerName="maas-api" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943959 4898 cpu_manager.go:414] "containerMap: removing container" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943963 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943970 4898 cpu_manager.go:414] "containerMap: removing container" podUID="840943c2-edc7-4b62-bb37-c1b5881a743b" containerName="maas-api" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943974 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="840943c2-edc7-4b62-bb37-c1b5881a743b" containerName="maas-api" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943984 4898 cpu_manager.go:414] "containerMap: removing container" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" Nov 03 23:10:56.944008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.943989 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944045 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944055 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="909a0d33-8e64-42a5-9ed0-67cb0a96a413" containerName="istio-proxy" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944062 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d915cf7-19ac-4e53-85d5-0b77359cdb93" containerName="maas-api" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944068 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="953b0740-f330-4f73-a1b6-a1f9718acd7c" containerName="main" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944075 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86c6c87-ccf4-433e-bf67-850af6e8ec57" containerName="istio-proxy" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944080 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="24779e64-aabc-4fa5-8701-79d557ea9c32" containerName="istio-proxy" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944087 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="85456a7b-db98-4291-b3ed-e01fa8d7052f" containerName="istio-proxy" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944092 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf69038-2c53-4767-ab1e-a0447f0af9dd" containerName="maas-api" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944098 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4f2a4d-50bb-4517-be66-a5002ae59735" containerName="istio-proxy" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944103 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="840943c2-edc7-4b62-bb37-c1b5881a743b" containerName="maas-api" Nov 03 23:10:56.944548 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.944110 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="78915b29-4a76-4c77-8735-e67fa51c3fc4" containerName="istio-proxy" Nov 03 23:10:56.948437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.948420 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:56.970283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.970251 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:56.970417 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.970295 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:56.970417 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.970369 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsk6\" (UniqueName: \"kubernetes.io/projected/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-kube-api-access-jnsk6\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:56.979735 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:56.979710 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd"] Nov 03 23:10:57.041773 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.041731 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:57.041988 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.041943 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:10:57.070996 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.070961 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:57.071185 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.071018 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:57.071185 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.071035 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsk6\" (UniqueName: \"kubernetes.io/projected/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-kube-api-access-jnsk6\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:57.071775 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.071751 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:57.073404 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.073382 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:57.132399 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.132366 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsk6\" (UniqueName: \"kubernetes.io/projected/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-kube-api-access-jnsk6\") pod \"maas-api-revoke-sub-3d42a3fc-99ch2\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:10:57.260631 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:10:57.260555 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:11:01.980909 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:01.980859 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" podUID="9687ae02-dd9b-4030-955a-1576db895e90" containerName="payload-pre-processing" containerID="cri-o://41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea" gracePeriod=25 Nov 03 23:11:02.234166 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.234101 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 23:11:02.321364 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.321324 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9687ae02-dd9b-4030-955a-1576db895e90-kube-api-access-gc84l\" (UniqueName: \"kubernetes.io/projected/9687ae02-dd9b-4030-955a-1576db895e90-kube-api-access-gc84l\") pod \"9687ae02-dd9b-4030-955a-1576db895e90\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " Nov 03 23:11:02.321570 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.321372 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9687ae02-dd9b-4030-955a-1576db895e90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9687ae02-dd9b-4030-955a-1576db895e90-plugins-config-volume\") pod \"9687ae02-dd9b-4030-955a-1576db895e90\" (UID: \"9687ae02-dd9b-4030-955a-1576db895e90\") " Nov 03 23:11:02.321812 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.321765 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9687ae02-dd9b-4030-955a-1576db895e90-plugins-config-volume" pod "9687ae02-dd9b-4030-955a-1576db895e90" (UID: "9687ae02-dd9b-4030-955a-1576db895e90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:02.323541 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.323518 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9687ae02-dd9b-4030-955a-1576db895e90-kube-api-access-gc84l" pod "9687ae02-dd9b-4030-955a-1576db895e90" (UID: "9687ae02-dd9b-4030-955a-1576db895e90"). InnerVolumeSpecName "kube-api-access-gc84l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:02.422992 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.422953 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc84l\" (UniqueName: \"kubernetes.io/projected/9687ae02-dd9b-4030-955a-1576db895e90-kube-api-access-gc84l\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:02.422992 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.422985 4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9687ae02-dd9b-4030-955a-1576db895e90-plugins-config-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:02.548737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.548638 4898 generic.go:356] "Generic (PLEG): container finished" podID="9687ae02-dd9b-4030-955a-1576db895e90" containerID="41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea" exitCode=0 Nov 03 23:11:02.548737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.548686 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" event={"ID":"9687ae02-dd9b-4030-955a-1576db895e90","Type":"ContainerDied","Data":"41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea"} Nov 03 23:11:02.548737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.548689 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" Nov 03 23:11:02.548737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.548728 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd" event={"ID":"9687ae02-dd9b-4030-955a-1576db895e90","Type":"ContainerDied","Data":"02220f0cef6b6080805c08baf380b80534cc8e20a8c093261ece5177b3373ad9"} Nov 03 23:11:02.549154 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.548750 4898 scope.go:122] "RemoveContainer" containerID="41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea" Nov 03 23:11:02.560125 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.560101 4898 scope.go:122] "RemoveContainer" containerID="41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea" Nov 03 23:11:02.560378 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:11:02.560357 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea\": container with ID starting with 41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea not found: ID does not exist" containerID="41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea" Nov 03 23:11:02.560441 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.560389 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea"} err="failed to get container status \"41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea\": rpc error: code = NotFound desc = could not find container \"41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea\": container with ID starting with 41246539e55ff5cabb8306597fde4ee10c46268c53502d3ee82b3355e4ee7cea not found: ID does not exist" Nov 03 23:11:02.716740 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.716702 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd"] Nov 03 23:11:02.720515 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.720491 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f1190-7568b9dd79-jvzsd"] Nov 03 23:11:02.904676 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:02.904638 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9687ae02-dd9b-4030-955a-1576db895e90" path="/var/lib/kubelet/pods/9687ae02-dd9b-4030-955a-1576db895e90/volumes" Nov 03 23:11:04.785431 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:04.785367 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" containerID="cri-o://66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884" gracePeriod=45 Nov 03 23:11:05.037628 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.037573 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 23:11:05.147682 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.147648 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34711dce-3ebd-4870-9431-f06db08ee855-kube-api-access-6s9gb\" (UniqueName: \"kubernetes.io/projected/34711dce-3ebd-4870-9431-f06db08ee855-kube-api-access-6s9gb\") pod \"34711dce-3ebd-4870-9431-f06db08ee855\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " Nov 03 23:11:05.147863 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.147716 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34711dce-3ebd-4870-9431-f06db08ee855-tls-certs\" (UniqueName: \"kubernetes.io/secret/34711dce-3ebd-4870-9431-f06db08ee855-tls-certs\") pod \"34711dce-3ebd-4870-9431-f06db08ee855\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " Nov 03 23:11:05.147863 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.147777 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-home\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-home\") pod \"34711dce-3ebd-4870-9431-f06db08ee855\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " Nov 03 23:11:05.147863 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.147816 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-dshm\") pod \"34711dce-3ebd-4870-9431-f06db08ee855\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " Nov 03 23:11:05.147863 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.147853 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-tmp-dir\") pod \"34711dce-3ebd-4870-9431-f06db08ee855\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " Nov 03 23:11:05.148056 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.147930 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-model-cache\") pod \"34711dce-3ebd-4870-9431-f06db08ee855\" (UID: \"34711dce-3ebd-4870-9431-f06db08ee855\") " Nov 03 23:11:05.148147 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.148121 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-home" pod "34711dce-3ebd-4870-9431-f06db08ee855" (UID: "34711dce-3ebd-4870-9431-f06db08ee855"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:05.148238 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.148155 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-tmp-dir" pod "34711dce-3ebd-4870-9431-f06db08ee855" (UID: "34711dce-3ebd-4870-9431-f06db08ee855"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:05.148354 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.148264 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-model-cache" pod "34711dce-3ebd-4870-9431-f06db08ee855" (UID: "34711dce-3ebd-4870-9431-f06db08ee855"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:05.148354 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.148295 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:05.148354 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.148315 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:05.150044 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.150015 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34711dce-3ebd-4870-9431-f06db08ee855-kube-api-access-6s9gb" pod "34711dce-3ebd-4870-9431-f06db08ee855" (UID: "34711dce-3ebd-4870-9431-f06db08ee855"). InnerVolumeSpecName "kube-api-access-6s9gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:05.150369 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.150344 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-dshm" pod "34711dce-3ebd-4870-9431-f06db08ee855" (UID: "34711dce-3ebd-4870-9431-f06db08ee855"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:05.150480 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.150392 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34711dce-3ebd-4870-9431-f06db08ee855-tls-certs" pod "34711dce-3ebd-4870-9431-f06db08ee855" (UID: "34711dce-3ebd-4870-9431-f06db08ee855"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:05.249729 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.249692 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s9gb\" (UniqueName: \"kubernetes.io/projected/34711dce-3ebd-4870-9431-f06db08ee855-kube-api-access-6s9gb\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:05.249729 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.249725 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34711dce-3ebd-4870-9431-f06db08ee855-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:05.249729 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.249735 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:05.249984 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.249744 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34711dce-3ebd-4870-9431-f06db08ee855-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:05.565579 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.565544 4898 generic.go:356] "Generic (PLEG): container finished" podID="34711dce-3ebd-4870-9431-f06db08ee855" containerID="66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884" exitCode=0 Nov 03 23:11:05.565818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.565596 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" Nov 03 23:11:05.565818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.565626 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" event={"ID":"34711dce-3ebd-4870-9431-f06db08ee855","Type":"ContainerDied","Data":"66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884"} Nov 03 23:11:05.565818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.565678 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9" event={"ID":"34711dce-3ebd-4870-9431-f06db08ee855","Type":"ContainerDied","Data":"f1808cb6930bf4bd258cdc03772263f17d067b0d972df11b2c1334508328b265"} Nov 03 23:11:05.565818 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.565702 4898 scope.go:122] "RemoveContainer" containerID="66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884" Nov 03 23:11:05.584247 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.584227 4898 scope.go:122] "RemoveContainer" containerID="66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884" Nov 03 23:11:05.584552 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:11:05.584531 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884\": container with ID starting with 66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884 not found: ID does not exist" containerID="66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884" Nov 03 23:11:05.584606 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.584562 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884"} err="failed to get container status \"66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884\": rpc error: code = NotFound desc = could not find container \"66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884\": container with ID starting with 66e96e8c3f6dae622dfd34702607fd23d1ed569e72ec175ad9ab54cd81d3f884 not found: ID does not exist" Nov 03 23:11:05.731016 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.730979 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9"] Nov 03 23:11:05.734699 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:05.734666 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6f1190/distinct-w5-6f1190-kserve-df5d8fc95-r7tj9"] Nov 03 23:11:06.905819 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:06.905770 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34711dce-3ebd-4870-9431-f06db08ee855" path="/var/lib/kubelet/pods/34711dce-3ebd-4870-9431-f06db08ee855/volumes" Nov 03 23:11:19.835049 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:11:19.834966 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:11:26.005005 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:26.004970 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2"] Nov 03 23:11:26.369020 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:26.368990 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2"] Nov 03 23:11:26.370956 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:11:26.370916 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1eaca2a_24ee_43d5_80fb_509508d8b3c9.slice/crio-8e8105bb993cd7e03ae46863cc1dec3d647a45f72a6fd43279890358f4270b0a WatchSource:0}: Error finding container 8e8105bb993cd7e03ae46863cc1dec3d647a45f72a6fd43279890358f4270b0a: Status 404 returned error can't find the container with id 8e8105bb993cd7e03ae46863cc1dec3d647a45f72a6fd43279890358f4270b0a Nov 03 23:11:26.671234 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:26.671203 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" event={"ID":"f1eaca2a-24ee-43d5-80fb-509508d8b3c9","Type":"ContainerStarted","Data":"3699824b5f7b54e0b2b5fd47316066742f7e9bb198e8078e2b3cbdd6afbb2fc0"} Nov 03 23:11:26.671387 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:26.671263 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" event={"ID":"f1eaca2a-24ee-43d5-80fb-509508d8b3c9","Type":"ContainerStarted","Data":"8e8105bb993cd7e03ae46863cc1dec3d647a45f72a6fd43279890358f4270b0a"} Nov 03 23:11:26.790136 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:26.790079 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" podStartSLOduration=30.790066541 podStartE2EDuration="30.790066541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:26.787956625 +0000 UTC m=+2930.000725676" watchObservedRunningTime="2026-11-03 23:11:26.790066541 +0000 UTC m=+2930.002835591" Nov 03 23:11:27.678830 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:27.678763 4898 generic.go:356] "Generic (PLEG): container finished" podID="f1eaca2a-24ee-43d5-80fb-509508d8b3c9" containerID="3699824b5f7b54e0b2b5fd47316066742f7e9bb198e8078e2b3cbdd6afbb2fc0" exitCode=0 Nov 03 23:11:27.679264 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:27.678850 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" event={"ID":"f1eaca2a-24ee-43d5-80fb-509508d8b3c9","Type":"ContainerDied","Data":"3699824b5f7b54e0b2b5fd47316066742f7e9bb198e8078e2b3cbdd6afbb2fc0"} Nov 03 23:11:28.825949 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.825925 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:11:28.874741 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.874710 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-kube-api-access-jnsk6\" (UniqueName: \"kubernetes.io/projected/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-kube-api-access-jnsk6\") pod \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " Nov 03 23:11:28.876768 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.876745 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-kube-api-access-jnsk6" pod "f1eaca2a-24ee-43d5-80fb-509508d8b3c9" (UID: "f1eaca2a-24ee-43d5-80fb-509508d8b3c9"). InnerVolumeSpecName "kube-api-access-jnsk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:28.975906 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.975822 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-openshift-ca\") pod \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " Nov 03 23:11:28.976056 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.975926 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-xks-ca\") pod \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\" (UID: \"f1eaca2a-24ee-43d5-80fb-509508d8b3c9\") " Nov 03 23:11:28.976179 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.976163 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsk6\" (UniqueName: \"kubernetes.io/projected/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-kube-api-access-jnsk6\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:28.976221 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.976167 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-openshift-ca" pod "f1eaca2a-24ee-43d5-80fb-509508d8b3c9" (UID: "f1eaca2a-24ee-43d5-80fb-509508d8b3c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:28.978074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:28.978049 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-xks-ca" pod "f1eaca2a-24ee-43d5-80fb-509508d8b3c9" (UID: "f1eaca2a-24ee-43d5-80fb-509508d8b3c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:29.076619 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:29.076583 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-openshift-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:29.076619 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:29.076618 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaca2a-24ee-43d5-80fb-509508d8b3c9-maas-api-xks-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:29.690051 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:29.690017 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" event={"ID":"f1eaca2a-24ee-43d5-80fb-509508d8b3c9","Type":"ContainerDied","Data":"8e8105bb993cd7e03ae46863cc1dec3d647a45f72a6fd43279890358f4270b0a"} Nov 03 23:11:29.690051 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:29.690042 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2" Nov 03 23:11:29.690230 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:11:29.690066 4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e8105bb993cd7e03ae46863cc1dec3d647a45f72a6fd43279890358f4270b0a" Nov 03 23:11:49.834160 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:11:49.834118 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:12:06.013403 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.013366 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b656c5c54-p66vm"] Nov 03 23:12:06.013897 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.013571 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b656c5c54-p66vm" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerName="manager" containerID="cri-o://bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2" gracePeriod=30 Nov 03 23:12:06.390684 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.390659 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 23:12:06.526055 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.526015 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-webhook-cert\") pod \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " Nov 03 23:12:06.526251 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.526146 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-kube-api-access-f9pss\" (UniqueName: \"kubernetes.io/projected/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-kube-api-access-f9pss\") pod \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " Nov 03 23:12:06.526251 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.526202 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-metrics-certs\") pod \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\" (UID: \"f7ff4bb6-8c87-4027-baf5-0a80a0b44261\") " Nov 03 23:12:06.528608 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.528576 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-kube-api-access-f9pss" pod "f7ff4bb6-8c87-4027-baf5-0a80a0b44261" (UID: "f7ff4bb6-8c87-4027-baf5-0a80a0b44261"). InnerVolumeSpecName "kube-api-access-f9pss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:12:06.529593 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.529548 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-metrics-certs" pod "f7ff4bb6-8c87-4027-baf5-0a80a0b44261" (UID: "f7ff4bb6-8c87-4027-baf5-0a80a0b44261"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:06.530113 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.530087 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-webhook-cert" pod "f7ff4bb6-8c87-4027-baf5-0a80a0b44261" (UID: "f7ff4bb6-8c87-4027-baf5-0a80a0b44261"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:06.627241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.627206 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9pss\" (UniqueName: \"kubernetes.io/projected/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-kube-api-access-f9pss\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:06.627241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.627234 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:06.627241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.627244 4898 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7ff4bb6-8c87-4027-baf5-0a80a0b44261-webhook-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:06.886310 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.886232 4898 generic.go:356] "Generic (PLEG): container finished" podID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerID="bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2" exitCode=0 Nov 03 23:12:06.886310 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.886281 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-p66vm" Nov 03 23:12:06.886505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.886316 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-p66vm" event={"ID":"f7ff4bb6-8c87-4027-baf5-0a80a0b44261","Type":"ContainerDied","Data":"bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2"} Nov 03 23:12:06.886505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.886380 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-p66vm" event={"ID":"f7ff4bb6-8c87-4027-baf5-0a80a0b44261","Type":"ContainerDied","Data":"00cd435ed2b03af9df2cb1c2fb5e68ce7735cee3382bf53ae953ba4aeee3bddb"} Nov 03 23:12:06.886505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.886412 4898 scope.go:122] "RemoveContainer" containerID="bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2" Nov 03 23:12:06.896568 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.896549 4898 scope.go:122] "RemoveContainer" containerID="0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6" Nov 03 23:12:06.905560 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.905543 4898 scope.go:122] "RemoveContainer" containerID="bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2" Nov 03 23:12:06.905828 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:12:06.905808 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2\": container with ID starting with bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2 not found: ID does not exist" containerID="bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2" Nov 03 23:12:06.905900 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.905835 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2"} err="failed to get container status \"bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2\": rpc error: code = NotFound desc = could not find container \"bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2\": container with ID starting with bf1e83656e1c9f3681d2156ec4495ef2bed45cf650b0feabc541433c870479e2 not found: ID does not exist" Nov 03 23:12:06.905900 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.905854 4898 scope.go:122] "RemoveContainer" containerID="0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6" Nov 03 23:12:06.906062 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:12:06.906048 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6\": container with ID starting with 0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6 not found: ID does not exist" containerID="0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6" Nov 03 23:12:06.906102 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:06.906064 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6"} err="failed to get container status \"0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6\": rpc error: code = NotFound desc = could not find container \"0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6\": container with ID starting with 0a9641d38234550e7fcfb21bf74b5a9de6262eed0f0d5d490d9c0149c5d67ab6 not found: ID does not exist" Nov 03 23:12:07.053673 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:07.053635 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b656c5c54-p66vm"] Nov 03 23:12:07.103807 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:07.103758 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b656c5c54-p66vm"] Nov 03 23:12:08.032712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.032680 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b656c5c54-6sr56"] Nov 03 23:12:08.033109 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033095 4898 cpu_manager.go:414] "containerMap: removing container" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" Nov 03 23:12:08.033156 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033110 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" Nov 03 23:12:08.033156 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033133 4898 cpu_manager.go:414] "containerMap: removing container" podUID="f1eaca2a-24ee-43d5-80fb-509508d8b3c9" containerName="revoke-keys" Nov 03 23:12:08.033156 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033138 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1eaca2a-24ee-43d5-80fb-509508d8b3c9" containerName="revoke-keys" Nov 03 23:12:08.033156 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033154 4898 cpu_manager.go:414] "containerMap: removing container" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerName="manager" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033159 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerName="manager" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033174 4898 cpu_manager.go:414] "containerMap: removing container" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerName="manager" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033179 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerName="manager" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033188 4898 cpu_manager.go:414] "containerMap: removing container" podUID="9687ae02-dd9b-4030-955a-1576db895e90" containerName="payload-pre-processing" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033193 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9687ae02-dd9b-4030-955a-1576db895e90" containerName="payload-pre-processing" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033246 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1eaca2a-24ee-43d5-80fb-509508d8b3c9" containerName="revoke-keys" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033259 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerName="manager" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033267 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9687ae02-dd9b-4030-955a-1576db895e90" containerName="payload-pre-processing" Nov 03 23:12:08.033283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.033273 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="34711dce-3ebd-4870-9431-f06db08ee855" containerName="main" Nov 03 23:12:08.038067 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.038040 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.139960 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.139925 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7559\" (UniqueName: \"kubernetes.io/projected/e5f075f1-db45-486e-ac69-6c0e45aaa92e-kube-api-access-j7559\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.140405 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.140046 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-metrics-certs\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.140405 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.140082 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-webhook-cert\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.141490 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.141374 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 23:12:08.141490 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.141459 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6fd48\"" Nov 03 23:12:08.141910 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.141868 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 23:12:08.142071 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.142056 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 23:12:08.143731 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.143710 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b656c5c54-6sr56"] Nov 03 23:12:08.241188 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.241142 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-metrics-certs\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.241393 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.241202 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-webhook-cert\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.241393 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.241274 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7559\" (UniqueName: \"kubernetes.io/projected/e5f075f1-db45-486e-ac69-6c0e45aaa92e-kube-api-access-j7559\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.265722 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.265657 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-webhook-cert\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.265722 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.265691 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-metrics-certs\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.302941 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.302870 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7559\" (UniqueName: \"kubernetes.io/projected/e5f075f1-db45-486e-ac69-6c0e45aaa92e-kube-api-access-j7559\") pod \"maas-controller-b656c5c54-6sr56\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.357223 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.357179 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:08.738079 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.738020 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b656c5c54-6sr56"] Nov 03 23:12:08.741864 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:12:08.741830 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f075f1_db45_486e_ac69_6c0e45aaa92e.slice/crio-c69fc00949f07b8df093bc7acb98962fa9785e5862737530fee6ac797dab6bad WatchSource:0}: Error finding container c69fc00949f07b8df093bc7acb98962fa9785e5862737530fee6ac797dab6bad: Status 404 returned error can't find the container with id c69fc00949f07b8df093bc7acb98962fa9785e5862737530fee6ac797dab6bad Nov 03 23:12:08.743219 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.743198 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:12:08.909235 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.909196 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" path="/var/lib/kubelet/pods/f7ff4bb6-8c87-4027-baf5-0a80a0b44261/volumes" Nov 03 23:12:08.909614 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:08.909590 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-6sr56" event={"ID":"e5f075f1-db45-486e-ac69-6c0e45aaa92e","Type":"ContainerStarted","Data":"c69fc00949f07b8df093bc7acb98962fa9785e5862737530fee6ac797dab6bad"} Nov 03 23:12:09.906660 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:09.906628 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-6sr56" event={"ID":"e5f075f1-db45-486e-ac69-6c0e45aaa92e","Type":"ContainerStarted","Data":"ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf"} Nov 03 23:12:09.907075 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:09.906710 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:10.025034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:10.024930 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b656c5c54-6sr56" podStartSLOduration=2.031986882 podStartE2EDuration="3.024916967s" totalImagesPullingTime="992.930085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:10.024673327 +0000 UTC m=+2973.237442379" watchObservedRunningTime="2026-11-03 23:12:10.024916967 +0000 UTC m=+2973.237686018" Nov 03 23:12:19.833551 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:12:19.833508 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:12:20.913370 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:20.913345 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:34.409806 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.409754 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx"] Nov 03 23:12:34.410465 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.410442 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ff4bb6-8c87-4027-baf5-0a80a0b44261" containerName="manager" Nov 03 23:12:34.413873 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.413850 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486570 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486537 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486738 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486577 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486738 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486595 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486738 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486672 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486882 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486755 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486882 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486855 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486948 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486887 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c610d1e-997c-4518-93d5-b5928ab5f99c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486948 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486910 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.486948 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.486933 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lf6\" (UniqueName: \"kubernetes.io/projected/8c610d1e-997c-4518-93d5-b5928ab5f99c-kube-api-access-r8lf6\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.587680 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.587641 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c610d1e-997c-4518-93d5-b5928ab5f99c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.587680 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.587682 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.587969 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.587704 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lf6\" (UniqueName: \"kubernetes.io/projected/8c610d1e-997c-4518-93d5-b5928ab5f99c-kube-api-access-r8lf6\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.587969 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.587758 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.587969 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.587783 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.587969 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.587839 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588190 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.587971 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588292 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.588215 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588292 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.588253 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588292 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.588286 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588459 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.588364 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588554 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.588524 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588678 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.588655 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c610d1e-997c-4518-93d5-b5928ab5f99c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.588762 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.588658 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.590096 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.590073 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.590365 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.590343 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.619454 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.619421 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx"] Nov 03 23:12:34.649590 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.649560 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c610d1e-997c-4518-93d5-b5928ab5f99c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.649751 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.649653 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lf6\" (UniqueName: \"kubernetes.io/projected/8c610d1e-997c-4518-93d5-b5928ab5f99c-kube-api-access-r8lf6\") pod \"maas-default-gateway-openshift-default-59f957fdd7-l9zcx\" (UID: \"8c610d1e-997c-4518-93d5-b5928ab5f99c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:34.727218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:34.727141 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:35.152645 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:35.152616 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx"] Nov 03 23:12:35.155958 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:12:35.154904 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c610d1e_997c_4518_93d5_b5928ab5f99c.slice/crio-8965465ecee513fae9122b37a26ece3c3e640e12a1333d16fbb8e521b0ac4215 WatchSource:0}: Error finding container 8965465ecee513fae9122b37a26ece3c3e640e12a1333d16fbb8e521b0ac4215: Status 404 returned error can't find the container with id 8965465ecee513fae9122b37a26ece3c3e640e12a1333d16fbb8e521b0ac4215 Nov 03 23:12:35.158373 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:35.158334 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:35.158478 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:35.158412 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:35.158478 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:35.158454 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:36.037814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:36.033968 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" event={"ID":"8c610d1e-997c-4518-93d5-b5928ab5f99c","Type":"ContainerStarted","Data":"c27f962be8c815dee56a63e1cf5529ad26cea3e4c0deaa26f4a84c7e106591d1"} Nov 03 23:12:36.037814 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:36.034032 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" event={"ID":"8c610d1e-997c-4518-93d5-b5928ab5f99c","Type":"ContainerStarted","Data":"8965465ecee513fae9122b37a26ece3c3e640e12a1333d16fbb8e521b0ac4215"} Nov 03 23:12:36.209944 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:36.209888 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" podStartSLOduration=2.209874721 podStartE2EDuration="2.209874721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:36.209356458 +0000 UTC m=+2999.422125528" watchObservedRunningTime="2026-11-03 23:12:36.209874721 +0000 UTC m=+2999.422643771" Nov 03 23:12:36.727723 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:36.727686 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:36.729220 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:36.729189 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-l9zcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 23:12:36.729356 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:36.729240 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" podUID="8c610d1e-997c-4518-93d5-b5928ab5f99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 23:12:37.578783 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:37.578753 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:12:37.583100 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:37.583078 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:12:37.679523 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:37.679497 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:12:37.683564 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:37.683540 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:12:37.727778 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:37.727748 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-l9zcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 23:12:37.727911 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:37.727821 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" podUID="8c610d1e-997c-4518-93d5-b5928ab5f99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 23:12:38.728056 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:38.728019 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-l9zcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 23:12:38.728454 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:38.728074 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" podUID="8c610d1e-997c-4518-93d5-b5928ab5f99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 23:12:39.728059 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:39.728025 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-l9zcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 23:12:39.728436 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:39.728097 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" podUID="8c610d1e-997c-4518-93d5-b5928ab5f99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 23:12:40.728518 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:40.728479 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-l9zcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 23:12:40.728952 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:40.728537 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" podUID="8c610d1e-997c-4518-93d5-b5928ab5f99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 23:12:41.727985 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:41.727948 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-l9zcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 23:12:41.728176 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:41.728000 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" podUID="8c610d1e-997c-4518-93d5-b5928ab5f99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 23:12:41.798091 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:41.798059 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b656c5c54-6sr56"] Nov 03 23:12:41.798533 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:41.798278 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b656c5c54-6sr56" podUID="e5f075f1-db45-486e-ac69-6c0e45aaa92e" containerName="manager" containerID="cri-o://ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf" gracePeriod=30 Nov 03 23:12:42.055493 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.055456 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:42.085196 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.085157 4898 generic.go:356] "Generic (PLEG): container finished" podID="e5f075f1-db45-486e-ac69-6c0e45aaa92e" containerID="ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf" exitCode=0 Nov 03 23:12:42.085196 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.085200 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-6sr56" Nov 03 23:12:42.085419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.085240 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-6sr56" event={"ID":"e5f075f1-db45-486e-ac69-6c0e45aaa92e","Type":"ContainerDied","Data":"ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf"} Nov 03 23:12:42.085419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.085293 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-6sr56" event={"ID":"e5f075f1-db45-486e-ac69-6c0e45aaa92e","Type":"ContainerDied","Data":"c69fc00949f07b8df093bc7acb98962fa9785e5862737530fee6ac797dab6bad"} Nov 03 23:12:42.085419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.085316 4898 scope.go:122] "RemoveContainer" containerID="ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf" Nov 03 23:12:42.093913 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.093896 4898 scope.go:122] "RemoveContainer" containerID="ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf" Nov 03 23:12:42.094169 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:12:42.094152 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf\": container with ID starting with ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf not found: ID does not exist" containerID="ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf" Nov 03 23:12:42.094216 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.094175 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf"} err="failed to get container status \"ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf\": rpc error: code = NotFound desc = could not find container \"ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf\": container with ID starting with ebc71ec4e994ad974a8bfb6be4b3fe4368bebe13766c99437173b5bb877a86bf not found: ID does not exist" Nov 03 23:12:42.176984 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.176946 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-webhook-cert\") pod \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " Nov 03 23:12:42.177275 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.177255 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-metrics-certs\") pod \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " Nov 03 23:12:42.177452 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.177435 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f075f1-db45-486e-ac69-6c0e45aaa92e-kube-api-access-j7559\" (UniqueName: \"kubernetes.io/projected/e5f075f1-db45-486e-ac69-6c0e45aaa92e-kube-api-access-j7559\") pod \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\" (UID: \"e5f075f1-db45-486e-ac69-6c0e45aaa92e\") " Nov 03 23:12:42.186052 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.185994 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-webhook-cert" pod "e5f075f1-db45-486e-ac69-6c0e45aaa92e" (UID: "e5f075f1-db45-486e-ac69-6c0e45aaa92e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:42.191525 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.191489 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f075f1-db45-486e-ac69-6c0e45aaa92e-kube-api-access-j7559" pod "e5f075f1-db45-486e-ac69-6c0e45aaa92e" (UID: "e5f075f1-db45-486e-ac69-6c0e45aaa92e"). InnerVolumeSpecName "kube-api-access-j7559". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:12:42.191659 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.191541 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-metrics-certs" pod "e5f075f1-db45-486e-ac69-6c0e45aaa92e" (UID: "e5f075f1-db45-486e-ac69-6c0e45aaa92e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:42.279200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.279168 4898 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-webhook-cert\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:42.279200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.279199 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f075f1-db45-486e-ac69-6c0e45aaa92e-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:42.279200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.279212 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7559\" (UniqueName: \"kubernetes.io/projected/e5f075f1-db45-486e-ac69-6c0e45aaa92e-kube-api-access-j7559\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:42.509844 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.509806 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b656c5c54-6sr56"] Nov 03 23:12:42.513604 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.513582 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b656c5c54-6sr56"] Nov 03 23:12:42.907408 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:42.907366 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f075f1-db45-486e-ac69-6c0e45aaa92e" path="/var/lib/kubelet/pods/e5f075f1-db45-486e-ac69-6c0e45aaa92e/volumes" Nov 03 23:12:43.424798 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.424765 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:43.426998 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.425258 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:43.426998 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.426068 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-l9zcx" Nov 03 23:12:43.905919 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.905885 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b656c5c54-2jdd6"] Nov 03 23:12:43.906306 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.906294 4898 cpu_manager.go:414] "containerMap: removing container" podUID="e5f075f1-db45-486e-ac69-6c0e45aaa92e" containerName="manager" Nov 03 23:12:43.906351 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.906307 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f075f1-db45-486e-ac69-6c0e45aaa92e" containerName="manager" Nov 03 23:12:43.906392 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.906380 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f075f1-db45-486e-ac69-6c0e45aaa92e" containerName="manager" Nov 03 23:12:43.909559 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.909542 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:43.965824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.965209 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 23:12:43.965824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.965220 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 23:12:43.965824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.965263 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 23:12:43.965824 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.965321 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6fd48\"" Nov 03 23:12:43.967569 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.967545 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b656c5c54-2jdd6"] Nov 03 23:12:43.995850 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.995817 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgj9n\" (UniqueName: \"kubernetes.io/projected/1f38ea2b-323e-4e55-a280-3989e0bc3d11-kube-api-access-sgj9n\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:43.995998 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.995909 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f38ea2b-323e-4e55-a280-3989e0bc3d11-metrics-certs\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:43.995998 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:43.995990 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f38ea2b-323e-4e55-a280-3989e0bc3d11-webhook-cert\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.096772 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.096734 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f38ea2b-323e-4e55-a280-3989e0bc3d11-webhook-cert\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.096978 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.096865 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgj9n\" (UniqueName: \"kubernetes.io/projected/1f38ea2b-323e-4e55-a280-3989e0bc3d11-kube-api-access-sgj9n\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.096978 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.096900 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f38ea2b-323e-4e55-a280-3989e0bc3d11-metrics-certs\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.099448 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.099418 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f38ea2b-323e-4e55-a280-3989e0bc3d11-webhook-cert\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.099564 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.099418 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f38ea2b-323e-4e55-a280-3989e0bc3d11-metrics-certs\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.158001 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.157939 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgj9n\" (UniqueName: \"kubernetes.io/projected/1f38ea2b-323e-4e55-a280-3989e0bc3d11-kube-api-access-sgj9n\") pod \"maas-controller-b656c5c54-2jdd6\" (UID: \"1f38ea2b-323e-4e55-a280-3989e0bc3d11\") " pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.222830 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.222775 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:44.595932 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:44.595906 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b656c5c54-2jdd6"] Nov 03 23:12:44.597569 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:12:44.597545 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f38ea2b_323e_4e55_a280_3989e0bc3d11.slice/crio-9fb7a6491057cb66048267f8f9324410af5fd64684180db16da7bc99a7b91c1b WatchSource:0}: Error finding container 9fb7a6491057cb66048267f8f9324410af5fd64684180db16da7bc99a7b91c1b: Status 404 returned error can't find the container with id 9fb7a6491057cb66048267f8f9324410af5fd64684180db16da7bc99a7b91c1b Nov 03 23:12:45.103452 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:45.103417 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-2jdd6" event={"ID":"1f38ea2b-323e-4e55-a280-3989e0bc3d11","Type":"ContainerStarted","Data":"9fb7a6491057cb66048267f8f9324410af5fd64684180db16da7bc99a7b91c1b"} Nov 03 23:12:46.111283 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:46.111243 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b656c5c54-2jdd6" event={"ID":"1f38ea2b-323e-4e55-a280-3989e0bc3d11","Type":"ContainerStarted","Data":"9d316df9aba05bdbb15c7969ef855abb554ce4d16fa7ae2c3e737b251e28b5ce"} Nov 03 23:12:46.111690 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:46.111363 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:12:46.229537 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:46.229484 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b656c5c54-2jdd6" podStartSLOduration=2.308331393 podStartE2EDuration="3.229470857s" totalImagesPullingTime="921.139464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:46.229042347 +0000 UTC m=+3009.441811411" watchObservedRunningTime="2026-11-03 23:12:46.229470857 +0000 UTC m=+3009.442239959" Nov 03 23:12:49.834536 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:12:49.834488 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:12:57.119147 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:12:57.119115 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b656c5c54-2jdd6" Nov 03 23:13:19.834244 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:13:19.834202 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:13:49.834110 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:13:49.834073 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:14:05.057863 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.057827 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84"] Nov 03 23:14:05.062839 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.062819 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.143760 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.143722 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.143950 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.143773 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.143950 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.143916 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.144053 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.143957 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebef512-d473-45a0-9994-aefb9ef64f99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.144053 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.144015 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.144053 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.144046 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.144170 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.144071 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.144170 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.144123 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrc2\" (UniqueName: \"kubernetes.io/projected/5ebef512-d473-45a0-9994-aefb9ef64f99-kube-api-access-jmrc2\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.144253 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.144197 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.169148 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.169112 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84"] Nov 03 23:14:05.244923 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.244887 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.244950 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.244978 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebef512-d473-45a0-9994-aefb9ef64f99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.244997 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245034 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245063 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245411 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245101 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrc2\" (UniqueName: \"kubernetes.io/projected/5ebef512-d473-45a0-9994-aefb9ef64f99-kube-api-access-jmrc2\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245411 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245152 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245411 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245177 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245411 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245328 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245664 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245643 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245727 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245676 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245782 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245743 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.245782 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.245756 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebef512-d473-45a0-9994-aefb9ef64f99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.247398 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.247372 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.247515 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.247420 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.306155 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.306126 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebef512-d473-45a0-9994-aefb9ef64f99-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.306294 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.306134 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrc2\" (UniqueName: \"kubernetes.io/projected/5ebef512-d473-45a0-9994-aefb9ef64f99-kube-api-access-jmrc2\") pod \"maas-default-gateway-openshift-default-59f957fdd7-lch84\" (UID: \"5ebef512-d473-45a0-9994-aefb9ef64f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.375957 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.375926 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:05.801603 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.801577 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84"] Nov 03 23:14:05.803171 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:14:05.803143 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebef512_d473_45a0_9994_aefb9ef64f99.slice/crio-bf8d8139988d7f36f734a657f1099ff87d26b6f7b0f4580a12fe3065bdddd4fb WatchSource:0}: Error finding container bf8d8139988d7f36f734a657f1099ff87d26b6f7b0f4580a12fe3065bdddd4fb: Status 404 returned error can't find the container with id bf8d8139988d7f36f734a657f1099ff87d26b6f7b0f4580a12fe3065bdddd4fb Nov 03 23:14:05.805241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.805210 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:14:05.805323 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.805272 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:14:05.805323 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:05.805303 4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:14:06.518713 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:06.518674 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" event={"ID":"5ebef512-d473-45a0-9994-aefb9ef64f99","Type":"ContainerStarted","Data":"fb454efc572aeea6a29b7bdb00b82638595e22f9cddc17ad51d1589bf292bc5a"} Nov 03 23:14:06.519117 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:06.518758 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" event={"ID":"5ebef512-d473-45a0-9994-aefb9ef64f99","Type":"ContainerStarted","Data":"bf8d8139988d7f36f734a657f1099ff87d26b6f7b0f4580a12fe3065bdddd4fb"} Nov 03 23:14:07.376297 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:07.376256 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:07.377751 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:07.377716 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 23:14:07.377908 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:07.377765 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 23:14:08.377331 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:08.377290 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 23:14:08.377756 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:08.377344 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 23:14:09.377302 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:09.377267 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 23:14:09.377480 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:09.377319 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 23:14:10.377290 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:10.377250 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 23:14:10.377480 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:10.377303 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 23:14:11.376543 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:11.376507 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 23:14:11.377074 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:11.376561 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 23:14:12.377263 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:12.377226 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 23:14:12.377672 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:12.377279 4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 23:14:14.069434 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:14.069398 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:14.070096 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:14.070073 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:14.070907 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:14.070870 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" Nov 03 23:14:14.241078 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:14.241027 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podStartSLOduration=9.241012645 podStartE2EDuration="9.241012645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:14:06.662652994 +0000 UTC m=+3089.875422139" watchObservedRunningTime="2026-11-03 23:14:14.241012645 +0000 UTC m=+3097.453781695" Nov 03 23:14:19.833542 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:14:19.833451 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:14:45.116006 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:45.115958 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:14:45.116540 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:14:45.116031 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:14:49.833727 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:14:49.833688 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:15:00.580085 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.580052 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6"] Nov 03 23:15:00.583834 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.583815 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.638916 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.638884 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6"] Nov 03 23:15:00.684261 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.684233 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.684405 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.684374 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.684449 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.684398 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvd6\" (UniqueName: \"kubernetes.io/projected/a5049e43-e855-4160-a0e2-421fa5b267b0-kube-api-access-dxvd6\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.687189 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.687172 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:15:00.687269 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.687207 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:15:00.786016 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.785980 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.786218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.786098 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.786218 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.786117 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvd6\" (UniqueName: \"kubernetes.io/projected/a5049e43-e855-4160-a0e2-421fa5b267b0-kube-api-access-dxvd6\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.786712 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.786685 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.788449 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.788429 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.847720 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.847642 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvd6\" (UniqueName: \"kubernetes.io/projected/a5049e43-e855-4160-a0e2-421fa5b267b0-kube-api-access-dxvd6\") pod \"maas-api-revoke-sub-046e1343-lhrv6\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:00.896637 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:00.896607 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:01.270266 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:01.270241 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6"] Nov 03 23:15:01.271763 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:15:01.271732 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5049e43_e855_4160_a0e2_421fa5b267b0.slice/crio-ce324e3cfd63649dc03c1d30dc0591c3ae8cfdab5db67abcfb48b8ea65495ef2 WatchSource:0}: Error finding container ce324e3cfd63649dc03c1d30dc0591c3ae8cfdab5db67abcfb48b8ea65495ef2: Status 404 returned error can't find the container with id ce324e3cfd63649dc03c1d30dc0591c3ae8cfdab5db67abcfb48b8ea65495ef2 Nov 03 23:15:01.811383 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:01.811297 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" event={"ID":"a5049e43-e855-4160-a0e2-421fa5b267b0","Type":"ContainerStarted","Data":"a75f98d4e08a65a9990c3f593b6e4efe18c27be3aef7ce9a02e81bdb8c55256a"} Nov 03 23:15:01.811383 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:01.811354 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" event={"ID":"a5049e43-e855-4160-a0e2-421fa5b267b0","Type":"ContainerStarted","Data":"ce324e3cfd63649dc03c1d30dc0591c3ae8cfdab5db67abcfb48b8ea65495ef2"} Nov 03 23:15:01.928071 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:01.928019 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" podStartSLOduration=1.9280062230000001 podStartE2EDuration="1.928006223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:01.927340438 +0000 UTC m=+3145.140109509" watchObservedRunningTime="2026-11-03 23:15:01.928006223 +0000 UTC m=+3145.140775273" Nov 03 23:15:02.817733 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:02.817693 4898 generic.go:356] "Generic (PLEG): container finished" podID="a5049e43-e855-4160-a0e2-421fa5b267b0" containerID="a75f98d4e08a65a9990c3f593b6e4efe18c27be3aef7ce9a02e81bdb8c55256a" exitCode=0 Nov 03 23:15:02.818190 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:02.817751 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" event={"ID":"a5049e43-e855-4160-a0e2-421fa5b267b0","Type":"ContainerDied","Data":"a75f98d4e08a65a9990c3f593b6e4efe18c27be3aef7ce9a02e81bdb8c55256a"} Nov 03 23:15:03.958078 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:03.958053 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:04.117536 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.117458 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-openshift-ca\") pod \"a5049e43-e855-4160-a0e2-421fa5b267b0\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " Nov 03 23:15:04.117684 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.117563 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-xks-ca\") pod \"a5049e43-e855-4160-a0e2-421fa5b267b0\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " Nov 03 23:15:04.117684 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.117602 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5049e43-e855-4160-a0e2-421fa5b267b0-kube-api-access-dxvd6\" (UniqueName: \"kubernetes.io/projected/a5049e43-e855-4160-a0e2-421fa5b267b0-kube-api-access-dxvd6\") pod \"a5049e43-e855-4160-a0e2-421fa5b267b0\" (UID: \"a5049e43-e855-4160-a0e2-421fa5b267b0\") " Nov 03 23:15:04.117917 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.117889 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-openshift-ca" pod "a5049e43-e855-4160-a0e2-421fa5b267b0" (UID: "a5049e43-e855-4160-a0e2-421fa5b267b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:04.119605 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.119576 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-xks-ca" pod "a5049e43-e855-4160-a0e2-421fa5b267b0" (UID: "a5049e43-e855-4160-a0e2-421fa5b267b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:04.119757 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.119737 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5049e43-e855-4160-a0e2-421fa5b267b0-kube-api-access-dxvd6" pod "a5049e43-e855-4160-a0e2-421fa5b267b0" (UID: "a5049e43-e855-4160-a0e2-421fa5b267b0"). InnerVolumeSpecName "kube-api-access-dxvd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:04.218437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.218399 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-xks-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:04.218437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.218431 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxvd6\" (UniqueName: \"kubernetes.io/projected/a5049e43-e855-4160-a0e2-421fa5b267b0-kube-api-access-dxvd6\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:04.218437 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.218440 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5049e43-e855-4160-a0e2-421fa5b267b0-maas-api-openshift-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:04.839278 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.839242 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" Nov 03 23:15:04.839455 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.839286 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6" event={"ID":"a5049e43-e855-4160-a0e2-421fa5b267b0","Type":"ContainerDied","Data":"ce324e3cfd63649dc03c1d30dc0591c3ae8cfdab5db67abcfb48b8ea65495ef2"} Nov 03 23:15:04.839455 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:04.839354 4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce324e3cfd63649dc03c1d30dc0591c3ae8cfdab5db67abcfb48b8ea65495ef2" Nov 03 23:15:11.427019 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.426982 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj"] Nov 03 23:15:11.427412 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.427402 4898 cpu_manager.go:414] "containerMap: removing container" podUID="a5049e43-e855-4160-a0e2-421fa5b267b0" containerName="revoke-keys" Nov 03 23:15:11.427461 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.427413 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5049e43-e855-4160-a0e2-421fa5b267b0" containerName="revoke-keys" Nov 03 23:15:11.427509 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.427498 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5049e43-e855-4160-a0e2-421fa5b267b0" containerName="revoke-keys" Nov 03 23:15:11.430579 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.430563 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.534569 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.534533 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:15:11.534767 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.534665 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:15:11.536596 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.536573 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj"] Nov 03 23:15:11.594878 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.594846 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4450f256-7d0a-4446-8829-881b91230029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.595047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.594911 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5gh\" (UniqueName: \"kubernetes.io/projected/4450f256-7d0a-4446-8829-881b91230029-kube-api-access-7s5gh\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.595151 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.595040 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4450f256-7d0a-4446-8829-881b91230029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.695817 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.695728 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5gh\" (UniqueName: \"kubernetes.io/projected/4450f256-7d0a-4446-8829-881b91230029-kube-api-access-7s5gh\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.695817 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.695772 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4450f256-7d0a-4446-8829-881b91230029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.696003 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.695891 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4450f256-7d0a-4446-8829-881b91230029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.696551 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.696522 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4450f256-7d0a-4446-8829-881b91230029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.698316 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.698292 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4450f256-7d0a-4446-8829-881b91230029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:11.757443 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:11.757407 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5gh\" (UniqueName: \"kubernetes.io/projected/4450f256-7d0a-4446-8829-881b91230029-kube-api-access-7s5gh\") pod \"maas-api-revoke-sub-d3fba056-qc9cj\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:12.043475 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:12.043394 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:12.416798 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:12.416756 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj"] Nov 03 23:15:12.417361 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:15:12.417336 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4450f256_7d0a_4446_8829_881b91230029.slice/crio-b35417211d0a4158ac65e67bca62d3fc39bd0d697433547cf3702fbe495fb19c WatchSource:0}: Error finding container b35417211d0a4158ac65e67bca62d3fc39bd0d697433547cf3702fbe495fb19c: Status 404 returned error can't find the container with id b35417211d0a4158ac65e67bca62d3fc39bd0d697433547cf3702fbe495fb19c Nov 03 23:15:12.881674 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:12.881633 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" event={"ID":"4450f256-7d0a-4446-8829-881b91230029","Type":"ContainerStarted","Data":"0d1f9dc0be23e871678ec5b1f6ec862ae5dfec033f9547a06a99c3840b68def0"} Nov 03 23:15:12.882194 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:12.881700 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" event={"ID":"4450f256-7d0a-4446-8829-881b91230029","Type":"ContainerStarted","Data":"b35417211d0a4158ac65e67bca62d3fc39bd0d697433547cf3702fbe495fb19c"} Nov 03 23:15:12.998809 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:12.998688 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" podStartSLOduration=1.998674151 podStartE2EDuration="1.998674151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:12.997833473 +0000 UTC m=+3156.210602542" watchObservedRunningTime="2026-11-03 23:15:12.998674151 +0000 UTC m=+3156.211443196" Nov 03 23:15:13.888471 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:13.888434 4898 generic.go:356] "Generic (PLEG): container finished" podID="4450f256-7d0a-4446-8829-881b91230029" containerID="0d1f9dc0be23e871678ec5b1f6ec862ae5dfec033f9547a06a99c3840b68def0" exitCode=0 Nov 03 23:15:13.888471 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:13.888474 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" event={"ID":"4450f256-7d0a-4446-8829-881b91230029","Type":"ContainerDied","Data":"0d1f9dc0be23e871678ec5b1f6ec862ae5dfec033f9547a06a99c3840b68def0"} Nov 03 23:15:15.407981 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.407942 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:15.474181 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.473347 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4450f256-7d0a-4446-8829-881b91230029-kube-api-access-7s5gh\" (UniqueName: \"kubernetes.io/projected/4450f256-7d0a-4446-8829-881b91230029-kube-api-access-7s5gh\") pod \"4450f256-7d0a-4446-8829-881b91230029\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " Nov 03 23:15:15.474181 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.473448 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4450f256-7d0a-4446-8829-881b91230029-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4450f256-7d0a-4446-8829-881b91230029-maas-api-openshift-ca\") pod \"4450f256-7d0a-4446-8829-881b91230029\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " Nov 03 23:15:15.474181 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.473510 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4450f256-7d0a-4446-8829-881b91230029-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4450f256-7d0a-4446-8829-881b91230029-maas-api-xks-ca\") pod \"4450f256-7d0a-4446-8829-881b91230029\" (UID: \"4450f256-7d0a-4446-8829-881b91230029\") " Nov 03 23:15:15.479038 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.478983 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4450f256-7d0a-4446-8829-881b91230029-maas-api-openshift-ca" pod "4450f256-7d0a-4446-8829-881b91230029" (UID: "4450f256-7d0a-4446-8829-881b91230029"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:15.502102 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.489910 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4450f256-7d0a-4446-8829-881b91230029-maas-api-xks-ca" pod "4450f256-7d0a-4446-8829-881b91230029" (UID: "4450f256-7d0a-4446-8829-881b91230029"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:15.505281 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.503950 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4450f256-7d0a-4446-8829-881b91230029-kube-api-access-7s5gh" pod "4450f256-7d0a-4446-8829-881b91230029" (UID: "4450f256-7d0a-4446-8829-881b91230029"). InnerVolumeSpecName "kube-api-access-7s5gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:15.592731 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.592587 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4450f256-7d0a-4446-8829-881b91230029-maas-api-openshift-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:15.592731 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.592632 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4450f256-7d0a-4446-8829-881b91230029-maas-api-xks-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:15.592731 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.592648 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5gh\" (UniqueName: \"kubernetes.io/projected/4450f256-7d0a-4446-8829-881b91230029-kube-api-access-7s5gh\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:15.911612 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.911520 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" event={"ID":"4450f256-7d0a-4446-8829-881b91230029","Type":"ContainerDied","Data":"b35417211d0a4158ac65e67bca62d3fc39bd0d697433547cf3702fbe495fb19c"} Nov 03 23:15:15.911979 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.911958 4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b35417211d0a4158ac65e67bca62d3fc39bd0d697433547cf3702fbe495fb19c" Nov 03 23:15:15.912148 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:15:15.911909 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj" Nov 03 23:15:19.833619 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:15:19.833579 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:15:49.833110 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:15:49.833062 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:16:19.833892 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:16:19.833854 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:16:29.075866 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:16:29.075834 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2"] Nov 03 23:16:29.126126 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:16:29.126089 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d42a3fc-99ch2"] Nov 03 23:16:30.905848 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:16:30.905810 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1eaca2a-24ee-43d5-80fb-509508d8b3c9" path="/var/lib/kubelet/pods/f1eaca2a-24ee-43d5-80fb-509508d8b3c9/volumes" Nov 03 23:16:40.326965 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:16:40.326922 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j6q6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:16:40.327474 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:16:40.326980 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j6q6l" podUID="a8441d29-d512-4219-af7f-3a27ccbfd3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:16:49.833531 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:16:49.833493 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:17:19.834112 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:17:19.834030 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:17:37.610116 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:17:37.610086 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:17:37.614505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:17:37.614479 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:17:37.711622 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:17:37.711594 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:17:37.716285 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:17:37.716263 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:17:39.810809 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:17:39.810769 4898 scope.go:122] "RemoveContainer" containerID="3699824b5f7b54e0b2b5fd47316066742f7e9bb198e8078e2b3cbdd6afbb2fc0" Nov 03 23:17:49.833627 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:17:49.833591 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:18:19.833739 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:18:19.833694 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:18:49.833733 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:18:49.833659 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:19:19.833574 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:19:19.833535 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:19:49.839972 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:19:49.839931 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:20:04.058561 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:04.058526 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6"] Nov 03 23:20:04.108864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:04.108820 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhrv6"] Nov 03 23:20:04.905228 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:04.905182 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5049e43-e855-4160-a0e2-421fa5b267b0" path="/var/lib/kubelet/pods/a5049e43-e855-4160-a0e2-421fa5b267b0/volumes" Nov 03 23:20:16.048557 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:16.048520 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj"] Nov 03 23:20:16.098745 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:16.098712 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qc9cj"] Nov 03 23:20:16.906035 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:16.905948 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4450f256-7d0a-4446-8829-881b91230029" path="/var/lib/kubelet/pods/4450f256-7d0a-4446-8829-881b91230029/volumes" Nov 03 23:20:19.834882 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:20:19.834841 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:20:41.000951 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.000916 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s"] Nov 03 23:20:41.001439 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.001339 4898 cpu_manager.go:414] "containerMap: removing container" podUID="4450f256-7d0a-4446-8829-881b91230029" containerName="revoke-keys" Nov 03 23:20:41.001439 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.001350 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4450f256-7d0a-4446-8829-881b91230029" containerName="revoke-keys" Nov 03 23:20:41.001439 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.001433 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4450f256-7d0a-4446-8829-881b91230029" containerName="revoke-keys" Nov 03 23:20:41.004255 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.004234 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.060561 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.060532 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s"] Nov 03 23:20:41.081071 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.081037 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72cc9cb6-98d5-4933-869a-d26712bbfd04-tls-certs\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.081236 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.081092 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-model-cache\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.081236 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.081206 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpw46\" (UniqueName: \"kubernetes.io/projected/72cc9cb6-98d5-4933-869a-d26712bbfd04-kube-api-access-tpw46\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.081327 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.081249 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-dshm\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.081413 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.081388 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-home\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.081505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.081484 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-tmp-dir\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.108819 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.108755 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1376dc\"/\"openshift-service-ca.crt\"" Nov 03 23:20:41.108819 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.108766 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1376dc\"/\"kube-root-ca.crt\"" Nov 03 23:20:41.108819 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.108818 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1376dc\"/\"default-dockercfg-bhw7w\"" Nov 03 23:20:41.109043 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.108858 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1376dc\"/\"simulator-single-main-1376dc-kserve-self-signed-certs\"" Nov 03 23:20:41.182356 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182325 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-tmp-dir\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.182531 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182365 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72cc9cb6-98d5-4933-869a-d26712bbfd04-tls-certs\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.182531 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182424 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-model-cache\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.182531 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182476 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpw46\" (UniqueName: \"kubernetes.io/projected/72cc9cb6-98d5-4933-869a-d26712bbfd04-kube-api-access-tpw46\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.182531 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182506 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-dshm\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.182704 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182594 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-home\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.182967 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182944 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-model-cache\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.183148 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.182997 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-tmp-dir\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.183201 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.183165 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-home\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.184825 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.184806 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-dshm\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.185118 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.185098 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72cc9cb6-98d5-4933-869a-d26712bbfd04-tls-certs\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.243922 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.243888 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpw46\" (UniqueName: \"kubernetes.io/projected/72cc9cb6-98d5-4933-869a-d26712bbfd04-kube-api-access-tpw46\") pod \"simulator-single-main-1376dc-kserve-579fc96998-rhm4s\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.326181 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.326146 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:41.701832 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.701772 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s"] Nov 03 23:20:41.705253 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:20:41.705225 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72cc9cb6_98d5_4933_869a_d26712bbfd04.slice/crio-cf7abcc81a692a7583d2eadafbec2ad23538145c7438421e12848060caea61a9 WatchSource:0}: Error finding container cf7abcc81a692a7583d2eadafbec2ad23538145c7438421e12848060caea61a9: Status 404 returned error can't find the container with id cf7abcc81a692a7583d2eadafbec2ad23538145c7438421e12848060caea61a9 Nov 03 23:20:41.745951 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:41.745913 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" event={"ID":"72cc9cb6-98d5-4933-869a-d26712bbfd04","Type":"ContainerStarted","Data":"cf7abcc81a692a7583d2eadafbec2ad23538145c7438421e12848060caea61a9"} Nov 03 23:20:42.752043 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:42.752002 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" event={"ID":"72cc9cb6-98d5-4933-869a-d26712bbfd04","Type":"ContainerStarted","Data":"65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c"} Nov 03 23:20:42.870678 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:42.870618 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" podStartSLOduration=2.870602941 podStartE2EDuration="2.870602941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:42.869053706 +0000 UTC m=+3486.081822780" watchObservedRunningTime="2026-11-03 23:20:42.870602941 +0000 UTC m=+3486.083371992" Nov 03 23:20:43.326570 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:43.326503 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:49.833800 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:20:49.833747 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:20:51.326894 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:51.326848 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:51.339585 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:51.339552 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:20:51.809426 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:20:51.809339 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:21:19.834125 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:21:19.834082 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:21:39.878219 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:21:39.878180 4898 scope.go:122] "RemoveContainer" containerID="a75f98d4e08a65a9990c3f593b6e4efe18c27be3aef7ce9a02e81bdb8c55256a" Nov 03 23:21:39.887347 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:21:39.887325 4898 scope.go:122] "RemoveContainer" containerID="0d1f9dc0be23e871678ec5b1f6ec862ae5dfec033f9547a06a99c3840b68def0" Nov 03 23:21:49.835108 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:21:49.835072 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:21:54.529137 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:21:54.529098 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s"] Nov 03 23:22:09.549568 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.549504 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" podUID="72cc9cb6-98d5-4933-869a-d26712bbfd04" containerName="main" containerID="cri-o://65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c" gracePeriod=45 Nov 03 23:22:09.815983 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.815958 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:22:09.935840 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.934735 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cc9cb6-98d5-4933-869a-d26712bbfd04-tls-certs\" (UniqueName: \"kubernetes.io/secret/72cc9cb6-98d5-4933-869a-d26712bbfd04-tls-certs\") pod \"72cc9cb6-98d5-4933-869a-d26712bbfd04\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " Nov 03 23:22:09.935840 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.934864 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-model-cache\") pod \"72cc9cb6-98d5-4933-869a-d26712bbfd04\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " Nov 03 23:22:09.935840 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.934891 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-dshm\") pod \"72cc9cb6-98d5-4933-869a-d26712bbfd04\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " Nov 03 23:22:09.935840 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.934908 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-home\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-home\") pod \"72cc9cb6-98d5-4933-869a-d26712bbfd04\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " Nov 03 23:22:09.935840 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.934984 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cc9cb6-98d5-4933-869a-d26712bbfd04-kube-api-access-tpw46\" (UniqueName: \"kubernetes.io/projected/72cc9cb6-98d5-4933-869a-d26712bbfd04-kube-api-access-tpw46\") pod \"72cc9cb6-98d5-4933-869a-d26712bbfd04\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " Nov 03 23:22:09.935840 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.935002 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-tmp-dir\") pod \"72cc9cb6-98d5-4933-869a-d26712bbfd04\" (UID: \"72cc9cb6-98d5-4933-869a-d26712bbfd04\") " Nov 03 23:22:09.936621 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.936342 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-model-cache" pod "72cc9cb6-98d5-4933-869a-d26712bbfd04" (UID: "72cc9cb6-98d5-4933-869a-d26712bbfd04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:09.938008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.937420 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-tmp-dir" pod "72cc9cb6-98d5-4933-869a-d26712bbfd04" (UID: "72cc9cb6-98d5-4933-869a-d26712bbfd04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:09.938008 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.937602 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-home" pod "72cc9cb6-98d5-4933-869a-d26712bbfd04" (UID: "72cc9cb6-98d5-4933-869a-d26712bbfd04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:09.944267 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.944239 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cc9cb6-98d5-4933-869a-d26712bbfd04-kube-api-access-tpw46" pod "72cc9cb6-98d5-4933-869a-d26712bbfd04" (UID: "72cc9cb6-98d5-4933-869a-d26712bbfd04"). InnerVolumeSpecName "kube-api-access-tpw46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:09.948932 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.948895 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cc9cb6-98d5-4933-869a-d26712bbfd04-tls-certs" pod "72cc9cb6-98d5-4933-869a-d26712bbfd04" (UID: "72cc9cb6-98d5-4933-869a-d26712bbfd04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:22:09.950433 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:09.950397 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-dshm" pod "72cc9cb6-98d5-4933-869a-d26712bbfd04" (UID: "72cc9cb6-98d5-4933-869a-d26712bbfd04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:10.036228 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.036188 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpw46\" (UniqueName: \"kubernetes.io/projected/72cc9cb6-98d5-4933-869a-d26712bbfd04-kube-api-access-tpw46\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:10.036395 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.036230 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:10.036395 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.036248 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72cc9cb6-98d5-4933-869a-d26712bbfd04-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:10.036395 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.036260 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:10.036395 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.036272 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:10.036395 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.036284 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72cc9cb6-98d5-4933-869a-d26712bbfd04-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:10.194890 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.194798 4898 generic.go:356] "Generic (PLEG): container finished" podID="72cc9cb6-98d5-4933-869a-d26712bbfd04" containerID="65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c" exitCode=0 Nov 03 23:22:10.194890 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.194859 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" Nov 03 23:22:10.194890 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.194876 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" event={"ID":"72cc9cb6-98d5-4933-869a-d26712bbfd04","Type":"ContainerDied","Data":"65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c"} Nov 03 23:22:10.195128 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.194920 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" event={"ID":"72cc9cb6-98d5-4933-869a-d26712bbfd04","Type":"ContainerDied","Data":"cf7abcc81a692a7583d2eadafbec2ad23538145c7438421e12848060caea61a9"} Nov 03 23:22:10.195128 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.194940 4898 scope.go:122] "RemoveContainer" containerID="65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c" Nov 03 23:22:10.204924 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.204902 4898 scope.go:122] "RemoveContainer" containerID="65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c" Nov 03 23:22:10.205182 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:22:10.205161 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c\": container with ID starting with 65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c not found: ID does not exist" containerID="65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c" Nov 03 23:22:10.205257 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.205192 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c"} err="failed to get container status \"65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c\": rpc error: code = NotFound desc = could not find container \"65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c\": container with ID starting with 65bb2f1f6e03ad0300bf807d7e0d8bd70155e105c56c36174105d48919fcfd8c not found: ID does not exist" Nov 03 23:22:10.359505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.359469 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s"] Nov 03 23:22:10.409943 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.409904 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s"] Nov 03 23:22:10.798161 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.798118 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-1376dc/simulator-single-main-1376dc-kserve-579fc96998-rhm4s" podUID="72cc9cb6-98d5-4933-869a-d26712bbfd04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.151:8000/ready\": context deadline exceeded" Nov 03 23:22:10.905717 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:10.905681 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cc9cb6-98d5-4933-869a-d26712bbfd04" path="/var/lib/kubelet/pods/72cc9cb6-98d5-4933-869a-d26712bbfd04/volumes" Nov 03 23:22:19.834866 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:22:19.834826 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:22:37.641296 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:37.641267 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:22:37.645979 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:37.645953 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:22:37.744395 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:37.744367 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:22:37.749354 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:37.749333 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:22:42.054720 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.054683 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx"] Nov 03 23:22:42.055241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.054900 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" podUID="37b16595-3519-44cd-ae29-0e25b5e30edf" containerName="manager" containerID="cri-o://402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0" gracePeriod=10 Nov 03 23:22:42.314367 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.314312 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 23:22:42.355870 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.355830 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37b16595-3519-44cd-ae29-0e25b5e30edf-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37b16595-3519-44cd-ae29-0e25b5e30edf-extensions-socket-volume\") pod \"37b16595-3519-44cd-ae29-0e25b5e30edf\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " Nov 03 23:22:42.356087 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.355901 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b16595-3519-44cd-ae29-0e25b5e30edf-kube-api-access-pxbc9\" (UniqueName: \"kubernetes.io/projected/37b16595-3519-44cd-ae29-0e25b5e30edf-kube-api-access-pxbc9\") pod \"37b16595-3519-44cd-ae29-0e25b5e30edf\" (UID: \"37b16595-3519-44cd-ae29-0e25b5e30edf\") " Nov 03 23:22:42.356206 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.356183 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b16595-3519-44cd-ae29-0e25b5e30edf-extensions-socket-volume" pod "37b16595-3519-44cd-ae29-0e25b5e30edf" (UID: "37b16595-3519-44cd-ae29-0e25b5e30edf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:42.356989 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.356958 4898 generic.go:356] "Generic (PLEG): container finished" podID="37b16595-3519-44cd-ae29-0e25b5e30edf" containerID="402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0" exitCode=0 Nov 03 23:22:42.357103 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.357017 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" Nov 03 23:22:42.357103 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.357024 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" event={"ID":"37b16595-3519-44cd-ae29-0e25b5e30edf","Type":"ContainerDied","Data":"402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0"} Nov 03 23:22:42.357103 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.357066 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx" event={"ID":"37b16595-3519-44cd-ae29-0e25b5e30edf","Type":"ContainerDied","Data":"e7c22bf58cd5feddc1fb4ade5b2d0e8a9a115c661c75d8bbac57570fb6144800"} Nov 03 23:22:42.357103 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.357087 4898 scope.go:122] "RemoveContainer" containerID="402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0" Nov 03 23:22:42.358112 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.358083 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b16595-3519-44cd-ae29-0e25b5e30edf-kube-api-access-pxbc9" pod "37b16595-3519-44cd-ae29-0e25b5e30edf" (UID: "37b16595-3519-44cd-ae29-0e25b5e30edf"). InnerVolumeSpecName "kube-api-access-pxbc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:42.369897 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.369878 4898 scope.go:122] "RemoveContainer" containerID="402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0" Nov 03 23:22:42.370165 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:22:42.370140 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0\": container with ID starting with 402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0 not found: ID does not exist" containerID="402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0" Nov 03 23:22:42.370235 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.370174 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0"} err="failed to get container status \"402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0\": rpc error: code = NotFound desc = could not find container \"402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0\": container with ID starting with 402626015c373f62352d41c5d95a1f88fd30012987080a1a518cecda8b4320f0 not found: ID does not exist" Nov 03 23:22:42.457592 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.457557 4898 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37b16595-3519-44cd-ae29-0e25b5e30edf-extensions-socket-volume\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:42.457592 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.457594 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxbc9\" (UniqueName: \"kubernetes.io/projected/37b16595-3519-44cd-ae29-0e25b5e30edf-kube-api-access-pxbc9\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:42.778226 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.778194 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx"] Nov 03 23:22:42.784139 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.784111 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr4wx"] Nov 03 23:22:42.904836 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:22:42.904803 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b16595-3519-44cd-ae29-0e25b5e30edf" path="/var/lib/kubelet/pods/37b16595-3519-44cd-ae29-0e25b5e30edf/volumes" Nov 03 23:22:49.835078 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:22:49.834967 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:23:19.834561 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:23:19.834475 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:23:45.583625 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.583583 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v"] Nov 03 23:23:45.584052 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.584037 4898 cpu_manager.go:414] "containerMap: removing container" podUID="72cc9cb6-98d5-4933-869a-d26712bbfd04" containerName="main" Nov 03 23:23:45.584097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.584053 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cc9cb6-98d5-4933-869a-d26712bbfd04" containerName="main" Nov 03 23:23:45.584097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.584064 4898 cpu_manager.go:414] "containerMap: removing container" podUID="37b16595-3519-44cd-ae29-0e25b5e30edf" containerName="manager" Nov 03 23:23:45.584097 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.584070 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b16595-3519-44cd-ae29-0e25b5e30edf" containerName="manager" Nov 03 23:23:45.584198 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.584147 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cc9cb6-98d5-4933-869a-d26712bbfd04" containerName="main" Nov 03 23:23:45.584198 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.584156 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b16595-3519-44cd-ae29-0e25b5e30edf" containerName="manager" Nov 03 23:23:45.586904 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.586886 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:45.636917 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.636882 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v"] Nov 03 23:23:45.682950 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.682918 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lmmc9\"" Nov 03 23:23:45.732030 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.731998 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01bcf89d-3426-44b7-abac-0617cb91508c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vcp5v\" (UID: \"01bcf89d-3426-44b7-abac-0617cb91508c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:45.732223 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.732159 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d68z\" (UniqueName: \"kubernetes.io/projected/01bcf89d-3426-44b7-abac-0617cb91508c-kube-api-access-4d68z\") pod \"kuadrant-operator-controller-manager-678d87766f-vcp5v\" (UID: \"01bcf89d-3426-44b7-abac-0617cb91508c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:45.832820 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.832751 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01bcf89d-3426-44b7-abac-0617cb91508c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vcp5v\" (UID: \"01bcf89d-3426-44b7-abac-0617cb91508c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:45.833019 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.832910 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d68z\" (UniqueName: \"kubernetes.io/projected/01bcf89d-3426-44b7-abac-0617cb91508c-kube-api-access-4d68z\") pod \"kuadrant-operator-controller-manager-678d87766f-vcp5v\" (UID: \"01bcf89d-3426-44b7-abac-0617cb91508c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:45.833226 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.833203 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01bcf89d-3426-44b7-abac-0617cb91508c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vcp5v\" (UID: \"01bcf89d-3426-44b7-abac-0617cb91508c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:45.896330 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.896257 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d68z\" (UniqueName: \"kubernetes.io/projected/01bcf89d-3426-44b7-abac-0617cb91508c-kube-api-access-4d68z\") pod \"kuadrant-operator-controller-manager-678d87766f-vcp5v\" (UID: \"01bcf89d-3426-44b7-abac-0617cb91508c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:45.900373 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:45.900344 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:46.326573 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:46.326547 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v"] Nov 03 23:23:46.328650 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:23:46.328619 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bcf89d_3426_44b7_abac_0617cb91508c.slice/crio-46a1955ec1f67ea33507c498153f7165ad2658406cad230852e130d18d87619b WatchSource:0}: Error finding container 46a1955ec1f67ea33507c498153f7165ad2658406cad230852e130d18d87619b: Status 404 returned error can't find the container with id 46a1955ec1f67ea33507c498153f7165ad2658406cad230852e130d18d87619b Nov 03 23:23:46.665623 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:46.665523 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" event={"ID":"01bcf89d-3426-44b7-abac-0617cb91508c","Type":"ContainerStarted","Data":"445c097271f82ab9c9a7b81b544a1dcceb3782395a192eb51e14196b9d6895dd"} Nov 03 23:23:46.665623 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:46.665586 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" event={"ID":"01bcf89d-3426-44b7-abac-0617cb91508c","Type":"ContainerStarted","Data":"46a1955ec1f67ea33507c498153f7165ad2658406cad230852e130d18d87619b"} Nov 03 23:23:46.665623 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:46.665610 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:23:46.786387 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:46.786338 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" podStartSLOduration=1.786325978 podStartE2EDuration="1.786325978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:23:46.785048736 +0000 UTC m=+3669.997817816" watchObservedRunningTime="2026-11-03 23:23:46.786325978 +0000 UTC m=+3669.999095026" Nov 03 23:23:49.834808 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:23:49.834763 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:23:57.673098 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:23:57.673064 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vcp5v" Nov 03 23:24:19.833600 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:24:19.833556 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:24:30.123969 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:24:30.123924 4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-lch84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:24:30.124461 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:24:30.123979 4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lch84" podUID="5ebef512-d473-45a0-9994-aefb9ef64f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:24:49.836707 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:24:49.836667 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:25:19.834458 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:25:19.834420 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:25:36.947428 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:36.947391 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff"] Nov 03 23:25:36.950756 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:36.950738 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.055034 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.054998 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e433fd\"/\"openshift-service-ca.crt\"" Nov 03 23:25:37.055241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.055068 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e433fd\"/\"default-dockercfg-pzccb\"" Nov 03 23:25:37.055241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.055076 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e433fd\"/\"kube-root-ca.crt\"" Nov 03 23:25:37.055241 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.055105 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e433fd\"/\"simulator-single-main-e433fd-kserve-self-signed-certs\"" Nov 03 23:25:37.056947 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.056927 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff"] Nov 03 23:25:37.078032 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.078001 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-model-cache\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.078209 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.078191 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-tmp-dir\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.078299 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.078226 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc77eda-1730-46f9-a642-b49204cc26c3-tls-certs\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.078370 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.078350 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4t6g\" (UniqueName: \"kubernetes.io/projected/dfc77eda-1730-46f9-a642-b49204cc26c3-kube-api-access-l4t6g\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.078409 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.078394 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-dshm\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.078476 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.078451 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-home\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.179761 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.179715 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-model-cache\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.179761 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.179763 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-tmp-dir\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.180057 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.179808 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc77eda-1730-46f9-a642-b49204cc26c3-tls-certs\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.180057 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.179884 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4t6g\" (UniqueName: \"kubernetes.io/projected/dfc77eda-1730-46f9-a642-b49204cc26c3-kube-api-access-l4t6g\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.180057 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.179918 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-dshm\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.180057 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.179943 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-home\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.180279 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.180238 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-tmp-dir\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.180334 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.180272 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-model-cache\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.180394 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.180375 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-home\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.182323 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.182295 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-dshm\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.182434 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.182403 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc77eda-1730-46f9-a642-b49204cc26c3-tls-certs\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.241813 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.241732 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4t6g\" (UniqueName: \"kubernetes.io/projected/dfc77eda-1730-46f9-a642-b49204cc26c3-kube-api-access-l4t6g\") pod \"simulator-single-main-e433fd-kserve-667747fbdc-ws6ff\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.274117 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.274075 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:37.658212 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:37.658182 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff"] Nov 03 23:25:37.659958 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:25:37.659930 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfc77eda_1730_46f9_a642_b49204cc26c3.slice/crio-23a35eb1893ecf48ab01224955c3e3ec21c0acdbb5ba7453f198c28225ee68e8 WatchSource:0}: Error finding container 23a35eb1893ecf48ab01224955c3e3ec21c0acdbb5ba7453f198c28225ee68e8: Status 404 returned error can't find the container with id 23a35eb1893ecf48ab01224955c3e3ec21c0acdbb5ba7453f198c28225ee68e8 Nov 03 23:25:38.259434 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:38.259395 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" event={"ID":"dfc77eda-1730-46f9-a642-b49204cc26c3","Type":"ContainerStarted","Data":"7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab"} Nov 03 23:25:38.259434 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:38.259439 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" event={"ID":"dfc77eda-1730-46f9-a642-b49204cc26c3","Type":"ContainerStarted","Data":"23a35eb1893ecf48ab01224955c3e3ec21c0acdbb5ba7453f198c28225ee68e8"} Nov 03 23:25:38.274764 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:38.274739 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:38.378343 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:38.378290 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" podStartSLOduration=2.378277089 podStartE2EDuration="2.378277089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:38.376459351 +0000 UTC m=+3781.589228419" watchObservedRunningTime="2026-11-03 23:25:38.378277089 +0000 UTC m=+3781.591046139" Nov 03 23:25:47.274929 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:47.274892 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:47.287813 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:47.287771 4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:47.322569 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:25:47.322543 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:25:49.834702 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:25:49.834658 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:26:19.834416 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:26:19.834328 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:26:32.685584 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:32.685546 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff"] Nov 03 23:26:47.705909 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:47.705868 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" podUID="dfc77eda-1730-46f9-a642-b49204cc26c3" containerName="main" containerID="cri-o://7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab" gracePeriod=45 Nov 03 23:26:47.963419 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:47.963361 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:26:48.062288 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062249 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-home\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-home\") pod \"dfc77eda-1730-46f9-a642-b49204cc26c3\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " Nov 03 23:26:48.062477 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062314 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc77eda-1730-46f9-a642-b49204cc26c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc77eda-1730-46f9-a642-b49204cc26c3-tls-certs\") pod \"dfc77eda-1730-46f9-a642-b49204cc26c3\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " Nov 03 23:26:48.062477 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062353 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-dshm\") pod \"dfc77eda-1730-46f9-a642-b49204cc26c3\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " Nov 03 23:26:48.062622 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062528 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-tmp-dir\") pod \"dfc77eda-1730-46f9-a642-b49204cc26c3\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " Nov 03 23:26:48.062622 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062557 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-model-cache\") pod \"dfc77eda-1730-46f9-a642-b49204cc26c3\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " Nov 03 23:26:48.062622 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062584 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-home" pod "dfc77eda-1730-46f9-a642-b49204cc26c3" (UID: "dfc77eda-1730-46f9-a642-b49204cc26c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:48.062622 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062592 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc77eda-1730-46f9-a642-b49204cc26c3-kube-api-access-l4t6g\" (UniqueName: \"kubernetes.io/projected/dfc77eda-1730-46f9-a642-b49204cc26c3-kube-api-access-l4t6g\") pod \"dfc77eda-1730-46f9-a642-b49204cc26c3\" (UID: \"dfc77eda-1730-46f9-a642-b49204cc26c3\") " Nov 03 23:26:48.062894 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062825 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-model-cache" pod "dfc77eda-1730-46f9-a642-b49204cc26c3" (UID: "dfc77eda-1730-46f9-a642-b49204cc26c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:48.062894 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.062837 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-tmp-dir" pod "dfc77eda-1730-46f9-a642-b49204cc26c3" (UID: "dfc77eda-1730-46f9-a642-b49204cc26c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:48.063209 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.063187 4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-tmp-dir\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:48.063315 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.063211 4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-model-cache\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:48.063315 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.063225 4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-home\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:48.064946 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.064917 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-dshm" pod "dfc77eda-1730-46f9-a642-b49204cc26c3" (UID: "dfc77eda-1730-46f9-a642-b49204cc26c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:48.064946 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.064932 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc77eda-1730-46f9-a642-b49204cc26c3-tls-certs" pod "dfc77eda-1730-46f9-a642-b49204cc26c3" (UID: "dfc77eda-1730-46f9-a642-b49204cc26c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:26:48.065222 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.065204 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc77eda-1730-46f9-a642-b49204cc26c3-kube-api-access-l4t6g" pod "dfc77eda-1730-46f9-a642-b49204cc26c3" (UID: "dfc77eda-1730-46f9-a642-b49204cc26c3"). InnerVolumeSpecName "kube-api-access-l4t6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:48.164506 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.164468 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4t6g\" (UniqueName: \"kubernetes.io/projected/dfc77eda-1730-46f9-a642-b49204cc26c3-kube-api-access-l4t6g\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:48.164506 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.164503 4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc77eda-1730-46f9-a642-b49204cc26c3-tls-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:48.164506 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.164512 4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc77eda-1730-46f9-a642-b49204cc26c3-dshm\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:48.629826 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.629765 4898 generic.go:356] "Generic (PLEG): container finished" podID="dfc77eda-1730-46f9-a642-b49204cc26c3" containerID="7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab" exitCode=0 Nov 03 23:26:48.630000 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.629842 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" Nov 03 23:26:48.630000 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.629852 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" event={"ID":"dfc77eda-1730-46f9-a642-b49204cc26c3","Type":"ContainerDied","Data":"7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab"} Nov 03 23:26:48.630000 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.629900 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff" event={"ID":"dfc77eda-1730-46f9-a642-b49204cc26c3","Type":"ContainerDied","Data":"23a35eb1893ecf48ab01224955c3e3ec21c0acdbb5ba7453f198c28225ee68e8"} Nov 03 23:26:48.630000 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.629918 4898 scope.go:122] "RemoveContainer" containerID="7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab" Nov 03 23:26:48.640630 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.640612 4898 scope.go:122] "RemoveContainer" containerID="7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab" Nov 03 23:26:48.640976 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:26:48.640953 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab\": container with ID starting with 7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab not found: ID does not exist" containerID="7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab" Nov 03 23:26:48.641082 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.640983 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab"} err="failed to get container status \"7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab\": rpc error: code = NotFound desc = could not find container \"7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab\": container with ID starting with 7d9872261b223c0fb1ae6b7aff2a26ce0f91bb140d818cdb31a3bfeeb3d63eab not found: ID does not exist" Nov 03 23:26:48.799189 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.799150 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff"] Nov 03 23:26:48.849451 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.849420 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e433fd/simulator-single-main-e433fd-kserve-667747fbdc-ws6ff"] Nov 03 23:26:48.905186 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:26:48.905111 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc77eda-1730-46f9-a642-b49204cc26c3" path="/var/lib/kubelet/pods/dfc77eda-1730-46f9-a642-b49204cc26c3/volumes" Nov 03 23:26:49.834044 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:26:49.834007 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:27:19.834500 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:27:19.834463 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:27:37.672979 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:27:37.672952 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:27:37.677638 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:27:37.677617 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:27:37.780762 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:27:37.780736 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:27:37.785601 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:27:37.785579 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:27:49.925616 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:27:49.925494 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:28:19.852039 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:28:19.851993 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:28:49.833468 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:28:49.833431 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:29:19.837867 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:29:19.837756 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:29:45.328508 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.328469 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-797d89b4d5-7546x"] Nov 03 23:29:45.328953 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.328909 4898 cpu_manager.go:414] "containerMap: removing container" podUID="dfc77eda-1730-46f9-a642-b49204cc26c3" containerName="main" Nov 03 23:29:45.328953 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.328920 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc77eda-1730-46f9-a642-b49204cc26c3" containerName="main" Nov 03 23:29:45.329029 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.328995 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc77eda-1730-46f9-a642-b49204cc26c3" containerName="main" Nov 03 23:29:45.334578 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.334543 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.389254 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.389218 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-797d89b4d5-7546x"] Nov 03 23:29:45.435119 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.435086 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-metrics-certs\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.435281 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.435136 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-maas-api-tls\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.435281 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.435158 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6fbda7-ba4e-4541-8234-0e06c884ec27-openshift-service-ca\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.435361 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.435308 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2zr\" (UniqueName: \"kubernetes.io/projected/ab6fbda7-ba4e-4541-8234-0e06c884ec27-kube-api-access-nq2zr\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.436935 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.436908 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9qhjs\"" Nov 03 23:29:45.437956 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.437936 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:29:45.438041 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.437974 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 23:29:45.536286 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.536247 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-maas-api-tls\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.536505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.536292 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6fbda7-ba4e-4541-8234-0e06c884ec27-openshift-service-ca\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.536505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.536341 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2zr\" (UniqueName: \"kubernetes.io/projected/ab6fbda7-ba4e-4541-8234-0e06c884ec27-kube-api-access-nq2zr\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.536505 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.536446 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-metrics-certs\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.537156 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.537125 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6fbda7-ba4e-4541-8234-0e06c884ec27-openshift-service-ca\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.539080 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.539051 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-metrics-certs\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.539191 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.539093 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-maas-api-tls\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.598214 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.598140 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2zr\" (UniqueName: \"kubernetes.io/projected/ab6fbda7-ba4e-4541-8234-0e06c884ec27-kube-api-access-nq2zr\") pod \"maas-api-797d89b4d5-7546x\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:45.649305 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:45.649269 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:46.025976 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:46.025947 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-797d89b4d5-7546x"] Nov 03 23:29:46.027364 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:29:46.027329 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6fbda7_ba4e_4541_8234_0e06c884ec27.slice/crio-f1e19edd3a7f0c9901d65a621adfb17b1c575cb355beb460cd4bbc739cac94ad WatchSource:0}: Error finding container f1e19edd3a7f0c9901d65a621adfb17b1c575cb355beb460cd4bbc739cac94ad: Status 404 returned error can't find the container with id f1e19edd3a7f0c9901d65a621adfb17b1c575cb355beb460cd4bbc739cac94ad Nov 03 23:29:46.028737 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:46.028719 4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:29:46.640363 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:46.640332 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" event={"ID":"ab6fbda7-ba4e-4541-8234-0e06c884ec27","Type":"ContainerStarted","Data":"f1e19edd3a7f0c9901d65a621adfb17b1c575cb355beb460cd4bbc739cac94ad"} Nov 03 23:29:47.646875 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:47.646838 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" event={"ID":"ab6fbda7-ba4e-4541-8234-0e06c884ec27","Type":"ContainerStarted","Data":"7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4"} Nov 03 23:29:47.647286 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:47.646935 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:47.766379 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:47.766329 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" podStartSLOduration=1.848028283 podStartE2EDuration="2.766312898s" totalImagesPullingTime="918.284615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:47.765348975 +0000 UTC m=+4030.978118043" watchObservedRunningTime="2026-11-03 23:29:47.766312898 +0000 UTC m=+4030.979081948" Nov 03 23:29:49.834524 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:29:49.834485 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:29:53.219493 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.219454 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-797d89b4d5-7546x"] Nov 03 23:29:53.220623 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.219677 4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" podUID="ab6fbda7-ba4e-4541-8234-0e06c884ec27" containerName="maas-api" containerID="cri-o://7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4" gracePeriod=30 Nov 03 23:29:53.226085 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.226029 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:53.269458 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.269430 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb"] Nov 03 23:29:53.273137 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.273110 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.313864 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.313839 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb"] Nov 03 23:29:53.416442 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.416394 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49547\" (UniqueName: \"kubernetes.io/projected/816f9a02-2e61-4303-a1d4-a299ec21b617-kube-api-access-49547\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.416658 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.416447 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816f9a02-2e61-4303-a1d4-a299ec21b617-openshift-service-ca\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.416658 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.416621 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816f9a02-2e61-4303-a1d4-a299ec21b617-metrics-certs\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.416823 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.416776 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816f9a02-2e61-4303-a1d4-a299ec21b617-maas-api-tls\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.480022 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.479961 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:53.517306 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.517278 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816f9a02-2e61-4303-a1d4-a299ec21b617-maas-api-tls\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.517459 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.517322 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49547\" (UniqueName: \"kubernetes.io/projected/816f9a02-2e61-4303-a1d4-a299ec21b617-kube-api-access-49547\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.517459 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.517341 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816f9a02-2e61-4303-a1d4-a299ec21b617-openshift-service-ca\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.517587 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.517473 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816f9a02-2e61-4303-a1d4-a299ec21b617-metrics-certs\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.517959 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.517930 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816f9a02-2e61-4303-a1d4-a299ec21b617-openshift-service-ca\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.519986 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.519961 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816f9a02-2e61-4303-a1d4-a299ec21b617-metrics-certs\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.520075 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.519970 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816f9a02-2e61-4303-a1d4-a299ec21b617-maas-api-tls\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.579515 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.579479 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49547\" (UniqueName: \"kubernetes.io/projected/816f9a02-2e61-4303-a1d4-a299ec21b617-kube-api-access-49547\") pod \"maas-api-5b4b4b5f89-dczwb\" (UID: \"816f9a02-2e61-4303-a1d4-a299ec21b617\") " pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.618756 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.618721 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-metrics-certs\") pod \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " Nov 03 23:29:53.618921 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.618816 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-maas-api-tls\") pod \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " Nov 03 23:29:53.618968 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.618940 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6fbda7-ba4e-4541-8234-0e06c884ec27-kube-api-access-nq2zr\" (UniqueName: \"kubernetes.io/projected/ab6fbda7-ba4e-4541-8234-0e06c884ec27-kube-api-access-nq2zr\") pod \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " Nov 03 23:29:53.619095 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.619068 4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab6fbda7-ba4e-4541-8234-0e06c884ec27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6fbda7-ba4e-4541-8234-0e06c884ec27-openshift-service-ca\") pod \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\" (UID: \"ab6fbda7-ba4e-4541-8234-0e06c884ec27\") " Nov 03 23:29:53.619366 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.619343 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6fbda7-ba4e-4541-8234-0e06c884ec27-openshift-service-ca" pod "ab6fbda7-ba4e-4541-8234-0e06c884ec27" (UID: "ab6fbda7-ba4e-4541-8234-0e06c884ec27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:29:53.619476 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.619457 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:53.619576 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.619556 4898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6fbda7-ba4e-4541-8234-0e06c884ec27-openshift-service-ca\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:53.620865 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.620843 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-metrics-certs" pod "ab6fbda7-ba4e-4541-8234-0e06c884ec27" (UID: "ab6fbda7-ba4e-4541-8234-0e06c884ec27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:29:53.621005 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.620985 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6fbda7-ba4e-4541-8234-0e06c884ec27-kube-api-access-nq2zr" pod "ab6fbda7-ba4e-4541-8234-0e06c884ec27" (UID: "ab6fbda7-ba4e-4541-8234-0e06c884ec27"). InnerVolumeSpecName "kube-api-access-nq2zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:29:53.621152 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.621133 4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-maas-api-tls" pod "ab6fbda7-ba4e-4541-8234-0e06c884ec27" (UID: "ab6fbda7-ba4e-4541-8234-0e06c884ec27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:29:53.681338 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.681299 4898 generic.go:356] "Generic (PLEG): container finished" podID="ab6fbda7-ba4e-4541-8234-0e06c884ec27" containerID="7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4" exitCode=0 Nov 03 23:29:53.681485 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.681451 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" event={"ID":"ab6fbda7-ba4e-4541-8234-0e06c884ec27","Type":"ContainerDied","Data":"7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4"} Nov 03 23:29:53.681557 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.681506 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" event={"ID":"ab6fbda7-ba4e-4541-8234-0e06c884ec27","Type":"ContainerDied","Data":"f1e19edd3a7f0c9901d65a621adfb17b1c575cb355beb460cd4bbc739cac94ad"} Nov 03 23:29:53.681557 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.681536 4898 scope.go:122] "RemoveContainer" containerID="7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4" Nov 03 23:29:53.681721 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.681704 4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797d89b4d5-7546x" Nov 03 23:29:53.708418 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.708388 4898 scope.go:122] "RemoveContainer" containerID="7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4" Nov 03 23:29:53.708747 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:29:53.708724 4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4\": container with ID starting with 7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4 not found: ID does not exist" containerID="7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4" Nov 03 23:29:53.709393 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.708766 4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4"} err="failed to get container status \"7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4\": rpc error: code = NotFound desc = could not find container \"7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4\": container with ID starting with 7c731e69214edeeefc6d37dee8e7dc2e451d6dc4fb7b5df6a95d31e38534bbe4 not found: ID does not exist" Nov 03 23:29:53.720299 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.720278 4898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-metrics-certs\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:53.720389 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.720300 4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6fbda7-ba4e-4541-8234-0e06c884ec27-maas-api-tls\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:53.720389 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.720311 4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2zr\" (UniqueName: \"kubernetes.io/projected/ab6fbda7-ba4e-4541-8234-0e06c884ec27-kube-api-access-nq2zr\") on node \"ip-10-0-139-173.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:53.855944 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.855908 4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-797d89b4d5-7546x"] Nov 03 23:29:53.860026 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.859990 4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-797d89b4d5-7546x"] Nov 03 23:29:53.997442 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:53.997419 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb"] Nov 03 23:29:53.999286 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:29:53.999256 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816f9a02_2e61_4303_a1d4_a299ec21b617.slice/crio-f79f75da73f5c067d255317eeae46b8cf8f50a0c9dc8e9375e98c0005ab1fefa WatchSource:0}: Error finding container f79f75da73f5c067d255317eeae46b8cf8f50a0c9dc8e9375e98c0005ab1fefa: Status 404 returned error can't find the container with id f79f75da73f5c067d255317eeae46b8cf8f50a0c9dc8e9375e98c0005ab1fefa Nov 03 23:29:54.688769 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:54.688729 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" event={"ID":"816f9a02-2e61-4303-a1d4-a299ec21b617","Type":"ContainerStarted","Data":"f79f75da73f5c067d255317eeae46b8cf8f50a0c9dc8e9375e98c0005ab1fefa"} Nov 03 23:29:54.905295 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:54.905260 4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab6fbda7-ba4e-4541-8234-0e06c884ec27" path="/var/lib/kubelet/pods/ab6fbda7-ba4e-4541-8234-0e06c884ec27/volumes" Nov 03 23:29:55.695479 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:55.695446 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" event={"ID":"816f9a02-2e61-4303-a1d4-a299ec21b617","Type":"ContainerStarted","Data":"3c96a3968c32f8daebd388d2045330d7156cce9171ca43a3549675a8368bffed"} Nov 03 23:29:55.695916 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:55.695580 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:29:55.814350 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:29:55.814284 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" podStartSLOduration=1.895472697 podStartE2EDuration="2.814264989s" totalImagesPullingTime="918.792292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:55.813519504 +0000 UTC m=+4039.026288574" watchObservedRunningTime="2026-11-03 23:29:55.814264989 +0000 UTC m=+4039.027034043" Nov 03 23:30:01.705807 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:30:01.705755 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b4b4b5f89-dczwb" Nov 03 23:30:19.835103 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:30:19.835065 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:30:49.833738 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:30:49.833697 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:31:19.839020 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:31:19.838979 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:31:49.833377 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:31:49.833315 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:32:19.835012 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:32:19.834916 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:32:37.709386 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:32:37.709354 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:32:37.714692 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:32:37.714670 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:32:37.816220 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:32:37.816190 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:32:37.820603 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:32:37.820582 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:32:49.834570 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:32:49.834520 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:33:19.834397 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:33:19.834350 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:33:49.884813 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:33:49.884380 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:34:19.837171 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:34:19.837132 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:34:49.833515 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:34:49.833476 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:35:02.701017 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:02.700982 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xnpgs_97e3746c-a9f4-42f2-8dc4-2b62340078df/manager/0.log" Nov 03 23:35:02.976678 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:02.976602 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zkkb6_cee24d57-c7bb-4335-8a5e-9cd5ef61ae26/manager/0.log" Nov 03 23:35:03.303626 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:03.303527 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7vp25_665e97db-2cb5-44e4-950f-b8aae83c0df0/manager/0.log" Nov 03 23:35:03.604132 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:03.604099 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b656c5c54-2jdd6_1f38ea2b-323e-4e55-a280-3989e0bc3d11/manager/0.log" Nov 03 23:35:04.218022 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:04.217987 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bxcld_a3a64a96-32b4-4535-b038-6610314c54e5/manager/0.log" Nov 03 23:35:05.321526 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:05.321491 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b4b4b5f89-dczwb_816f9a02-2e61-4303-a1d4-a299ec21b617/maas-api/0.log" Nov 03 23:35:06.597304 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:06.597274 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qsvkc_a0861fcf-b07c-4b37-a7be-05855af26583/postgres/0.log" Nov 03 23:35:08.635917 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:08.635884 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-272st_3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e/manager/1.log" Nov 03 23:35:08.887859 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:08.887761 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-czcw4_8ae1270c-80be-4215-b91e-c5cb1d0462c1/manager/0.log" Nov 03 23:35:09.370309 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:09.370279 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vcp5v_01bcf89d-3426-44b7-abac-0617cb91508c/manager/0.log" Nov 03 23:35:10.122445 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:10.122410 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9w4hw_3ee52919-e882-49c4-967c-3a60a0632a47/manager/0.log" Nov 03 23:35:10.565912 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:10.565892 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-272st_3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e/manager/0.log" Nov 03 23:35:12.941200 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:12.941170 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-j6q6l_a8441d29-d512-4219-af7f-3a27ccbfd3fa/istio-proxy/0.log" Nov 03 23:35:13.265937 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:13.265855 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-l9zcx_8c610d1e-997c-4518-93d5-b5928ab5f99c/istio-proxy/0.log" Nov 03 23:35:13.538879 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:13.538744 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-lch84_5ebef512-d473-45a0-9994-aefb9ef64f99/istio-proxy/0.log" Nov 03 23:35:19.833970 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:35:19.833893 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:35:25.520198 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:35:25.520162 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74d647c79-nsnqv_700b9d15-342a-46d1-b94e-6b44aae2d9b5/router/0.log" Nov 03 23:35:49.834870 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:35:49.834823 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:36:18.909130 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:18.909098 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xnpgs_97e3746c-a9f4-42f2-8dc4-2b62340078df/manager/0.log" Nov 03 23:36:19.190076 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:19.189996 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zkkb6_cee24d57-c7bb-4335-8a5e-9cd5ef61ae26/manager/0.log" Nov 03 23:36:19.488191 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:19.488107 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7vp25_665e97db-2cb5-44e4-950f-b8aae83c0df0/manager/0.log" Nov 03 23:36:19.754118 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:19.754037 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b656c5c54-2jdd6_1f38ea2b-323e-4e55-a280-3989e0bc3d11/manager/0.log" Nov 03 23:36:19.834556 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:36:19.834524 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:36:20.251874 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:20.251835 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bxcld_a3a64a96-32b4-4535-b038-6610314c54e5/manager/0.log" Nov 03 23:36:21.210742 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:21.210712 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b4b4b5f89-dczwb_816f9a02-2e61-4303-a1d4-a299ec21b617/maas-api/0.log" Nov 03 23:36:22.487613 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:22.487576 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qsvkc_a0861fcf-b07c-4b37-a7be-05855af26583/postgres/0.log" Nov 03 23:36:24.178125 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:24.178095 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-272st_3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e/manager/1.log" Nov 03 23:36:24.434129 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:24.434037 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-czcw4_8ae1270c-80be-4215-b91e-c5cb1d0462c1/manager/0.log" Nov 03 23:36:24.897263 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:24.897229 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vcp5v_01bcf89d-3426-44b7-abac-0617cb91508c/manager/0.log" Nov 03 23:36:25.591510 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:25.591485 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9w4hw_3ee52919-e882-49c4-967c-3a60a0632a47/manager/0.log" Nov 03 23:36:27.874545 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:27.874512 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-j6q6l_a8441d29-d512-4219-af7f-3a27ccbfd3fa/istio-proxy/0.log" Nov 03 23:36:28.184032 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:28.183955 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-l9zcx_8c610d1e-997c-4518-93d5-b5928ab5f99c/istio-proxy/0.log" Nov 03 23:36:28.437620 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:28.437535 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-lch84_5ebef512-d473-45a0-9994-aefb9ef64f99/istio-proxy/0.log" Nov 03 23:36:30.300047 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:36:30.300015 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74d647c79-nsnqv_700b9d15-342a-46d1-b94e-6b44aae2d9b5/router/0.log" Nov 03 23:36:49.834328 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:36:49.834282 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:37:10.051693 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:37:10.051656 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b656c5c54-2jdd6_1f38ea2b-323e-4e55-a280-3989e0bc3d11/manager/0.log" Nov 03 23:37:10.601941 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:37:10.601906 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b656c5c54-2jdd6_1f38ea2b-323e-4e55-a280-3989e0bc3d11/manager/0.log" Nov 03 23:37:19.834427 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:37:19.834385 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:37:37.740535 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:37:37.740502 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:37:37.745297 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:37:37.745273 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:37:37.847187 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:37:37.847157 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-thjpw_e3a1808a-9488-4d49-8028-4860660c0c9d/console-operator/2.log" Nov 03 23:37:37.851458 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:37:37.851435 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhx44_724a681a-c911-439e-aa2c-904081578713/ovn-acl-logging/0.log" Nov 03 23:37:49.834250 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:37:49.834210 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:38:16.754173 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:16.754136 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b656c5c54-2jdd6_1f38ea2b-323e-4e55-a280-3989e0bc3d11/manager/0.log" Nov 03 23:38:17.316120 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:17.316086 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b656c5c54-2jdd6_1f38ea2b-323e-4e55-a280-3989e0bc3d11/manager/0.log" Nov 03 23:38:19.836378 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:38:19.836271 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:38:27.924339 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:27.924305 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c9btg_06b3993b-eaa8-4649-9c38-2b9bb7a22498/global-pull-secret-syncer/0.log" Nov 03 23:38:29.070201 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:29.070168 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6ddbh_e3939775-883b-454d-9e3b-6949f6984dfb/connection-checker/0.log" Nov 03 23:38:29.464222 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:29.464148 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hdlkm_35ea235a-c0ba-44b9-909f-e3dd472a6c36/connection-checker/0.log" Nov 03 23:38:29.866404 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:29.866375 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xrnfv_e0a3e17d-abf7-4d4b-afe0-fd028d2caeff/connection-checker/0.log" Nov 03 23:38:31.240377 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:31.240345 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pvpqf_c1a374ff-7146-47f5-bcbd-6c4dc710d580/konnectivity-agent/0.log" Nov 03 23:38:32.146854 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:32.146815 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-173.us-west-2.compute.internal_3dade66e2d9f1769c6a8531aeeaf63ad/haproxy/0.log" Nov 03 23:38:37.880227 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.880186 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-272st_3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e/manager/1.log" Nov 03 23:38:37.884280 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.884248 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-272st_3ec61a1c-adac-4fa3-ac3a-b2cb06f1058e/manager/0.log" Nov 03 23:38:37.900267 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.900238 4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq"] Nov 03 23:38:37.900701 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.900687 4898 cpu_manager.go:414] "containerMap: removing container" podUID="ab6fbda7-ba4e-4541-8234-0e06c884ec27" containerName="maas-api" Nov 03 23:38:37.900755 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.900703 4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab6fbda7-ba4e-4541-8234-0e06c884ec27" containerName="maas-api" Nov 03 23:38:37.900804 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.900779 4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab6fbda7-ba4e-4541-8234-0e06c884ec27" containerName="maas-api" Nov 03 23:38:37.903910 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.903894 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:37.956578 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.956536 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-md998\"/\"kube-root-ca.crt\"" Nov 03 23:38:37.997866 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.997834 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjcq\" (UniqueName: \"kubernetes.io/projected/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-kube-api-access-jnjcq\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:37.998060 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.998019 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-lib-modules\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:37.998140 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.998061 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-proc\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:37.998140 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.998105 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-podres\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:37.998263 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:37.998213 4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-sys\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.006765 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.006733 4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-md998\"/\"default-dockercfg-sg8mv\"" Nov 03 23:38:38.006765 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.006736 4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-md998\"/\"openshift-service-ca.crt\"" Nov 03 23:38:38.009120 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.009095 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq"] Nov 03 23:38:38.098764 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098726 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-lib-modules\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.098764 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098763 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-proc\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.099048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098819 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-podres\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.099048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098856 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-sys\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.099048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098896 4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjcq\" (UniqueName: \"kubernetes.io/projected/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-kube-api-access-jnjcq\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.099048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098912 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-lib-modules\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.099048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098926 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-proc\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.099048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098941 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-podres\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.099048 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.098959 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-sys\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.160650 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.160579 4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjcq\" (UniqueName: \"kubernetes.io/projected/cd5cefc1-4814-4f3f-aa41-6bcbee6820cc-kube-api-access-jnjcq\") pod \"perf-node-gather-daemonset-c4qmq\" (UID: \"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.216743 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.216713 4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:38.396772 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.396731 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-czcw4_8ae1270c-80be-4215-b91e-c5cb1d0462c1/manager/0.log" Nov 03 23:38:38.592904 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:38.592875 4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq"] Nov 03 23:38:38.594457 ip-10-0-139-173 kubenswrapper[4898]: W1103 23:38:38.594424 4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcd5cefc1_4814_4f3f_aa41_6bcbee6820cc.slice/crio-90da546e4dc632e95fab8d898894f061886714a88d0201c18ccbb4fcc1266ef6 WatchSource:0}: Error finding container 90da546e4dc632e95fab8d898894f061886714a88d0201c18ccbb4fcc1266ef6: Status 404 returned error can't find the container with id 90da546e4dc632e95fab8d898894f061886714a88d0201c18ccbb4fcc1266ef6 Nov 03 23:38:39.267222 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:39.267187 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vcp5v_01bcf89d-3426-44b7-abac-0617cb91508c/manager/0.log" Nov 03 23:38:39.362090 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:39.362045 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" event={"ID":"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc","Type":"ContainerStarted","Data":"058a6a74ccc7e434621f76ef86b377ba5b2d6e54c8832bbcfe6adffcff8dc4f2"} Nov 03 23:38:39.362090 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:39.362095 4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" event={"ID":"cd5cefc1-4814-4f3f-aa41-6bcbee6820cc","Type":"ContainerStarted","Data":"90da546e4dc632e95fab8d898894f061886714a88d0201c18ccbb4fcc1266ef6"} Nov 03 23:38:39.362362 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:39.362126 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:39.479612 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:39.479560 4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" podStartSLOduration=2.479546433 podStartE2EDuration="2.479546433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:39.478255619 +0000 UTC m=+4562.691024670" watchObservedRunningTime="2026-11-03 23:38:39.479546433 +0000 UTC m=+4562.692315493" Nov 03 23:38:40.177626 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:40.177587 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9w4hw_3ee52919-e882-49c4-967c-3a60a0632a47/manager/0.log" Nov 03 23:38:45.376853 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:45.376820 4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-md998/perf-node-gather-daemonset-c4qmq" Nov 03 23:38:45.693762 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:45.693678 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-s7gxg_60be4f8c-7051-444d-ad87-7e7895c94542/cluster-monitoring-operator/0.log" Nov 03 23:38:49.834864 ip-10-0-139-173 kubenswrapper[4898]: E1103 23:38:49.834823 4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r8md2" containerName="extractor" Nov 03 23:38:50.871943 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:50.871915 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vchhc_037aef30-2cc0-42c0-83bc-b685b2409e60/node-exporter/0.log" Nov 03 23:38:51.280555 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:51.280461 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vchhc_037aef30-2cc0-42c0-83bc-b685b2409e60/kube-rbac-proxy/0.log" Nov 03 23:38:51.690770 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:51.690743 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vchhc_037aef30-2cc0-42c0-83bc-b685b2409e60/init-textfile/0.log" Nov 03 23:38:56.293468 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:56.293437 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kqz7p_45e72fcb-4f35-48b5-8d07-efd0bda7809a/prometheus-operator/0.log" Nov 03 23:38:56.746347 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:56.746309 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kqz7p_45e72fcb-4f35-48b5-8d07-efd0bda7809a/kube-rbac-proxy/0.log" Nov 03 23:38:57.153713 ip-10-0-139-173 kubenswrapper[4898]: I1103 23:38:57.153686 4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-wc6kk_fce14729-dea1-47e8-9d4c-c4aa3b54130b/prometheus-operator-admission-webhook/0.log"